Functional Programming Unit Testing - Part 3
In the previous post, we talked about using QuickCheck as opposed to traditional xUnit framework testing. This covered some of the arguments for using property-based testing for our pure algorithms and relegating the xUnit framework tests for the rest. The question then arises, how do we tie it all together?
But, before we continue, let's get caught up to where we are today:
Integrating QuickCheck and xUnit frameworks
From the previous posts, we've talked about running the xUnit framework code and then running our QuickCheck property checks and both using separate frameworks. My goal is to break down that barrier so that I can have this as part of a continuous integration and be able to run all of my tests at once and have an easy to use mechanism for determining success or failure.
In the Haskell world, how might we do this?
Integration HUnit and QuickCheck with Test-Framework
In Haskell, we have two separate frameworks for testing, a traditional xUnit framework in HUnit and a property based testing framework in QuickCheck. Luckily, there is a framework out there to integrate the two of them together in a rather seamless manner called Test-Framework. With this tool, we can group our tests together into a list or two, depending on how you want them grouped. For example, I could take these following four QuickCheck and one HUnit tests together.
prop_rot13_equals s =
not (null s) ==> rot13 s == rot13 s
-- Single is inequal to original
prop_rot13_single_notEquals s =
not (null s) ==> rot13 s /= s
-- Double is equal to original
prop_rot13_double_equals s =
not (null s) ==> (rot13 . rot13) s == s
-- Distribution shapes should be equal
prop_rot13_group_equals s =
not (null s) ==> getDistro s == getDistro (rot13 s)
where getDistro = sort . map length . group . sort
test_rot13_empty =
assertEqual "rot13 should be empty" [] (rot13 [])
And then I could use Test-Framework in order to execute these as a group:
import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Encryption
tests =
[
testGroup "ROT13 Tests" [
testProperty "prop_rot13_equals" prop_rot13_equals,
testProperty "prop_rot13_single_notEquals" prop_rot13_single_notEquals,
testProperty "prop_rot13_double_equals" prop_rot13_double_equals,
testProperty "prop_rot13_group_equals" prop_rot13_group_equals,
testCase "test_rot13_empty" test_rot13_empty]
]
main = defaultMain tests
We can now run our tests from the GHCi window and see that our five tests pass.
prop_rot13_equals: [OK, passed 100 tests]
prop_rot13_single_notEquals: [OK, passed 100 tests]
prop_rot13_double_equals: [OK, passed 100 tests]
prop_rot13_group_equals: [OK, passed 100 tests]
test_rot13_empty: [OK]
Properties Test Cases Total
Passed 4 1 5
Failed 0 0 0
Total 4 1 5
As you can see from the above test output, we have a nice way of verifying our results for both our test cases and property-based checks in a cohesive runner. We can then integrate this into our continuous integration processes for running all of our tests.
In the F# world, what are my options?
Integrating FsCheck with Xunit.net
In a post entitled "F# + TestDriven.NET + xUnit.net = WIN", I showed a simple integration with F#, TestDriven.NET and xUnit.net that gives an entire integration story within Visual Studio. This time, let's take it a step further to include FsCheck, which is an implementation of QuickCheck 1.0 from the Haskell world. In order to make this happen, we need to create an IRunner interface instance using an object expression, that I talked about here.
namespace CodeBetter.Samples
module FsCheckExtensions =
open System
open FsCheck.Runner
open Xunit
let xUnitRunner =
{ new IRunner with
member x.OnArguments(ntest,args,every) =
args |>
List.iter(function
| :? IDisposable as d -> d.Dispose()
| _ -> ())
member x.OnFinished(name, result) =
match result with
| True data ->
Assert.True(true)
data.Stamps
|> Seq.iter (fun x -> printfn "%d - %A" (fst x) (snd x))
| False (_,args,None) ->
Assert.True(false, sprintf "%s - Falsifiable: %A" name args)
| False (_,args,Some exc) ->
Assert.True(false,
sprintf "%s - Falsifiable: %A with exception %O" name args exc)
| Exhausted data ->
Assert.True(false,
sprintf "Exhausted after %d tests" (data.NumberOfTests) )
}
let config = {quick with runner = xUnitRunner}
What this allows me to do is integrate the FsCheck runner with xUnit.net. As you can see, there are several conditions we must handle such as True, False with or without exception, and Exhausted. Implementing each is rather simple using the Assert.True function and passing in the given arguments. I think Exhausted and having an inconclusive state is not proper, so I add that as a failure instead. Other xUnit based frameworks have the Assert.Fail and Assert.Inconclusive and the latter I don't like because you should know what you're testing to know whether it succeeded or not.
Now that our runner is written, we can then integrate our property-based tests into using xUnit.net with this example. We'll use our ROT13 implementation again with four tests. In order to integrate with xUnit.net, we need to call the check function with our given config as we have above, and our property-based test.
not (List.is_empty s) ==>
propl (rot13 s = rot13 s)
Then we can define the actual test that xUnit.net will execute using a function decorated with a FactAttribute such as the following:
let test_prop_rot13_equals() =
check config prop_rot13_equals
Running that through the TestDriven.NET runner, we'll find that our code does indeed pass with flying colors. In addition, we could add additional tests that don't use property-based tests to the solution and have them run at the same time such as the following:
let rot13_emptyList_should_be_empty() =
Assert.Empty(rot13 [])
Below is the full implementation of the four tests in addition to the startup code required.
namespace CodeBetter.Samples
module EncryptionTests =
open System
open FsCheck
open FsCheck.Generator
open Xunit
// Local imports
open Encryption
open ListExtensions
open FsCheckExtensions
type CharGenerator() =
static member Chars = elements(['A'..'Z'] @ ['a'..'z'])
overwriteGenerators (typeof<CharGenerator>)
let prop_rot13_equals s =
not (List.is_empty s) ==>
propl (rot13 s = rot13 s)
[<Fact>]
let test_prop_rot13_equals() =
check config prop_rot13_equals
let prop_rot13_double_equals s =
not (List.is_empty s) ==>
propl ((rot13 >> rot13) s = s)
[<Fact>]
let test_prop_rot13_double_equals() =
check config prop_rot13_double_equals
let prop_rot13_single_notEquals s =
not (List.is_empty s) ==>
propl (rot13 s <> s)
[<Fact>]
let test_prop_rot13_single_notEquals() =
check config prop_rot13_single_notEquals
let prop_rot13_group_equals s =
let getDistro = ListExtensions.defaultSort >>
ListExtensions.group >>
List.map List.length >>
ListExtensions.defaultSort
not (List.is_empty s) ==>
propl (getDistro s = getDistro (rot13 s))
[<Fact>]
let test_prop_rot13_group_equals() =
check config prop_rot13_group_equals
[<Fact>]
let rot13_emptyList_should_be_empty() =
Assert.Empty(rot13 [])
When using the runner, we'll notice that all five tests pass in a relatively short order. It's a very nice integration. You may notice the ListExtensions module that I'm using in my prop_rot13_group_equals function. Basically, it's a port of some of the Haskell functions I use often and translated into F#. Some of these include sort, group, groupBy, span, sort, dropWhile, etc. I'll cover the full details in another post, but in the mean time, here are the implementations:
module ListExtensions =
open System
let defaultSort (l:#IComparable list) =
l |> List.sort (fun a b -> a.CompareTo(b))
let rec span (f:'a -> bool) (l:'a list) =
match f, l with
| _, [] -> [], []
| p, (x::xs' as xs) ->
let ys, zs = span p xs'
if p x then x::ys, zs else [], xs
let rec groupBy (f:'a -> 'a -> bool) (l:'a list) =
match f, l with
| _, [] -> []
| eq, x::xs ->
let ys, zs = span (eq x) xs
(x::ys) :: groupBy eq zs
let group (l:'a list) = groupBy (=) l
Pretty simple, yet powerful additions to the list implementations, in my opinion. After implementing this bridge between FsCheck and xUnit.net, we can then build this easily into our CI process.
Conclusion
In the functional programming world, we have two main ways of testing our code, either through the traditional xUnit tests or the more powerful QuickCheck property-based tests. Each of these are powerful in their own right, but made more powerful when combined into a single unit. When combined we have the power of integrating them into our CI process through some of the build/package tools in our tool belt.
