Writing the tests for FluentPath

Writing the tests for FluentPath is a challenge. The library is a wrapper around a legacy API (System.IO) that wasn’t designed to be easily testable.

If it were more testable, the sensible testing methodology would be to tell System.IO to act against a mock file system, which would enable me to verify that my code is doing the expected file system operations without having to manipulate the actual, physical file system: what we are testing here is FluentPath, not System.IO.

Unfortunately, that is not an option as nothing in System.IO enables us to plug a mock file system in. As a consequence, we are left with few options. A few people have suggested me to abstract my calls to System.IO away so that I could tell FluentPath – not System.IO – to use a mock instead of the real thing.

That in turn is getting a little silly: FluentPath already is a thin abstraction around System.IO, so layering another abstraction between them would double the test surface while bringing little or no value. I would have to test that new abstraction layer, and that would bring us back to square one.

Unless I’m missing something, the only option I have here is to bite the bullet and test against the real file system. Of course, the tests that do that can hardly be called unit tests. They are more integration tests as they don’t only test bits of my code. They really test the successful integration of my code with the underlying System.IO.

In order to write such tests, the techniques of BDD work particularly well as they enable you to express scenarios in natural language, from which test code is generated. Integration tests are being better expressed as scenarios orchestrating a few basic behaviors, so this is a nice fit.

The Orchard team has been successfully using SpecFlow for integration tests for a while and I thought it was pretty cool so that’s what I decided to use.

Consider for example the following scenario:

Scenario: Change extension
    Given a clean test directory
    When I change the extension of bar\notes.txt to foo
    Then bar\notes.txt should not exist
    And bar\notes.foo should exist

This is human readable and tells you everything you need to know about what you’re testing, but it is also executable code.

What happens when SpecFlow compiles this scenario is that it executes a bunch of regular expressions that identify the known Given (set-up phases), When (actions) and Then (result assertions) to identify the code to run, which is then translated into calls into the appropriate methods. Nothing magical. Here is the code generated by SpecFlow:

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Change extension")]
public virtual void ChangeExtension() {
  TechTalk.SpecFlow.ScenarioInfo scenarioInfo =
    new TechTalk.SpecFlow.ScenarioInfo("Change extension",
      ((string[])(null)));
  #line 6
  this.ScenarioSetup(scenarioInfo);
  #line 7
  testRunner.Given("a clean test directory");
  #line 8
  testRunner.When("I change the extension of " +
    "bar\\notes.txt to foo");
  #line 9
  testRunner.Then("bar\\notes.txt should not exist");
  #line 10
  testRunner.And("bar\\notes.foo should exist");
  #line hidden
  testRunner.CollectScenarioErrors();
}

The #line directives are there to give clues to the debugger, because yes, you can put breakpoints into a scenario:

The way you usually write tests with SpecFlow is that you write the scenario first, let it fail, then write the translation of your Given, When and Then into code if they don’t already exist, which results in running but failing tests, and then you write the code to make your tests pass (you implement the scenario).

In the case of FluentPath, I built a simple Given method that builds a simple file hierarchy in a temporary directory that all scenarios are going to work with:

[Given("a clean test directory")]
public void GivenACleanDirectory() {
  _path = new Path(SystemIO.Path.GetTempPath())
          .CreateSubDirectory("FluentPathSpecs")
          .MakeCurrent();
  _path.GetFileSystemEntries()
       .Delete(true);
  _path.CreateFile("foo.txt",
    "This is a text file named foo.");
  var bar = _path.CreateSubDirectory("bar");
  bar.CreateFile("baz.txt", "bar baz")
     .SetLastWriteTime(DateTime.Now.AddSeconds(-2));
  bar.CreateFile("notes.txt",
    "This is a text file containing notes.");
  var barbar = bar.CreateSubDirectory("bar");
  barbar.CreateFile("deep.txt", "Deep thoughts");
  var sub = _path.CreateSubDirectory("sub");
  sub.CreateSubDirectory("subsub");
  sub.CreateFile("baz.txt", "sub baz")
     .SetLastWriteTime(DateTime.Now);
  sub.CreateFile("binary.bin",
    new byte[] {0x00, 0x01, 0x02, 0x03,
                0x04, 0x05, 0xFF});
}

Then, to implement the scenario that you can read above, I had to write the following When:

[When("I change the extension of (.*) to (.*)")]
public void WhenIChangeTheExtension(
  string path, string newExtension) {
    var oldPath = Path.Current.Combine(path.Split('\\'));
    oldPath.Move(p => p.ChangeExtension(newExtension));
}

As you can see, the When attribute is specifying the regular expression that will enable the SpecFlow engine to recognize what When method to call and also how to map its parameters. For our scenario, “bar\notes.txt” will get mapped to the path parameter, and “foo” to the newExtension parameter.

And of course, the code that verifies the assumptions of the scenario:

[Then("(.*) should exist")]
public void ThenEntryShouldExist(string path) {
  Assert.IsTrue(_path.Combine(path.Split('\\')).Exists);
}

[Then("(.*) should not exist")]
public void ThenEntryShouldNotExist(string path) {
  Assert.IsFalse(_path.Combine(path.Split('\\')).Exists);
}

These steps should be written with reusability in mind. They are building blocks for your scenarios, not implementation of a specific scenario. Think small and fine-grained. In the case of the above steps, I could reuse each of those steps in other scenarios.

Those tests are easy to write and easier to read, which means that they also constitute a form of documentation.

Oh, and SpecFlow is just one way to do this. Rob wrote a long time ago about this sort of thing (but using a different framework) and I highly recommend this post if I somehow managed to pique your interest:
http://blog.wekeroad.com/blog/make-bdd-your-bff-2/

And this screencast (Rob always makes excellent screencasts):
http://blog.wekeroad.com/mvc-storefront/kona-3/
(click the “Download it here” link)

Finally, Rob (him again) tells me he did a free TekPub screencast on Specflow:
http://tekpub.com/view/concepts/5