Contents tagged with Frameworks
-
Exploring MapReduce with F#
With my exploration into mass concurrency and big data problems, I’m always finding challenges to give myself on how I might solve a given issue. Such examples that have intrigued me along the way as PLINQ, MPI/.NET, Data Parallel Haskell, but one in particular has intrigued me more – MapReduce. A challenge I gave myself is to fully understand this paradigm and implement a version using F#.
-
Language Oriented Programming and Functional Unit Testing in F#
As I've covered earlier, I'm very interested in the unit testing and behavior testing story in F# and functional programming. And as I've indicated earlier, I'm pretty fascinated by Domain Specific Languages (DSLs) as well in this regard. In the past, I've posted about some of the interesting things that are coming from the DSL community, especially from the .NET community itself. I wanted to challenge myself to prove that F# can also produce very readable and powerful DSLs as well. I think with the language flexibility that F# offers, I should be able to get quite expressive
Starting with DSLs
In part of what has me fascinated by DSLs lately is the recent book activity from both Martin Fowler and Ayende. Martin has been posting his Domain Specific Languages Work In Progress and keeping us somewhat up to date on its progress. So far I've been rather impressed with the book and I hope it continues. Although I think fluent builders are nice, I don't think they sometimes have the expressiveness of DSLs that I'm looking for. Many of the samples are written in both Java and C#, and it gives you a pretty good idea where he's going. However, I don't find the intent to be as clear though as DSLs written in Ruby, Boo or F#.
Ayende on the other hand has been working on his books "DSLs in Boo" that is partially available online right now. It's been interesting to see how flexible the language is in regards to DSLs and seeing it twist in new ways is pretty interesting. The first chaper is available for free, so you can check it out for yourself.
But with this, I was seeing such opportunities when I've been experimenting with language oriented programmiing and F#. Where are there opportunities?
Language Oriented Programming and FsUnit
I recently came across a dormant project on Google Code called FsUnit, which was a syntactic extension to NUnit using the F# language semantics. This to me looked like a great example of language oriented programming with F# and some of its uses. There isn't much to this library, yet very powerful. Let's take a look at a few samples:
1 |> should (equal 1)
"two" |> should (notEqual "three")
[| 1 .. 10 |] |> should (contain 5)
What makes this possible and pretty easy to understand? That would be the use of the forward operator. I've covered this operator in the past, but I can't emphasize its importance in this regard. If you're not familiar with that post, let's go over the basics. The basic definition of this operator is that it's just function application in reverse.
The forward operator has a basic definition that looks like this:
let (|>) x f = f x
What this allows us to do is put the first argument last. Let's take a quick example of just a simple forward operator:
let newList = [1..10] |> List.map(fun x -> x + x)
What it essentially becomes is the following code:
let newList = List.map(fun x -> x + x ) [1..10]
This gives us a couple of advantages doing this which includes:
- Intent Clarity - Allows you to perform data transformations, iterations, filtering, etc in a forward chaining fashion
- Type Inference - Since we're chaining together methods, we can better type infer when we specify the data first to the functions that will flow across all of them.
Just as an aside, Chris Smith, of the F# team, follows up from his recent presentation on Language Oriented Programming with F#. It's well worth the time invested to look at the possibilities that F# as a language offers. I am also hoping to do some presentations on this in the future as there is a lot of ground to cover with this language.
Replicating with C#?
During some of my recent talks on functional programming and F#, I have come across some resistance to the language itself because many saw that they can replicate a lot of the behaviors in C# 3.0. Indeed that is the case with many things, but not so much with where I'm going. One of my many gripes with C# and in turn the BCL is the fact that they don't treat Void (or Unit to those F#'ers) as a first class citizen. Therefore, I'm stuck with either using a Func<TArg, TResult> delegate or the Action<T> delegate for many operations and never the two shall combine. Whereas in F#, I'm free to create FastFunc<TArg, TResult> delegates with a Unit return type which returns nothing. It's a thing of beauty and I didn't have to mangle my code to get there.
So, with that in mind, let's see if we can replicate the forward operator in C# to the extent that I did in F#. The concept isn't really hard at all when you think about it. We're just calling a function with the arguments reversed. Let's create one with an actual return type, because remember, we're forced into the paradigm of either returning something or not with C# and we must use different delegates for it.
public static TResult ForwardFunc<TArg1, TArg2, TResult>(this TArg1 arg1, Func<TArg1, TArg2, TResult> func, TArg2 arg2)
{
return func(arg1, arg2);
}
var result = 4.ForwardFunc((x, y) => x * y, 3);
What I've been able to do is not as clean as F#, but a little interesting. What I'm doing is taking the first argument, then a function to process it, and then finally pass in my second argument. Below that function is a quick and dirty example of how to use it. But what about those times I don't want to return a value? Well, we have an answer for that as well:
public static void ForwardAction<TArg1, TArg2>(this TArg1 arg1, Action<TArg1, TArg2> func, TArg2 arg2)
{
func(arg1, arg2);
}
Enumerable.Range(1, 10).ForwardAction((x, y) => x.ShouldContain(y), 3);
What I did is change the Func<TArg1, TArg2, TResult> to a simple Action<TArg1, TArg2> which has the same effect. From there, I was able to use the xUnit.net 3.5 extensions to show that the IEnumerable<T> contained the number 3. If you mix in currying to this equation things get interesting quickly. I hope those who attended my functional C# class got some inkling, but since then I've been exploring more aspects of it as well as lessons learned. But the overall story is that it's just not as expressive as F# and kind of clumsy.
Extending the Syntax
As you may see, there really isn't much to FsUnit. Instead, I would like to apply some of those lessons to xUnit.net instead because of allowing static tests and it better fits my needs at the moment. Instead of relying on the underlying unit testing framework to take care of a lot of the details, instead, I'm going to use functional programming aspects of currying to make my DSL look more readable.
The first piece of the puzzle that we need is the basis of getting started. Let's put down the should and a few housekeeping items:
#light
namespace FsXUnit
#R @"D:\Tools\xunit-1.0.1\xunit.dll"
module Extensions =
open Xunit
let should f actual =
f actual
What I have created is a module under the FsXUnit namespace called Extensions which will hold my methods. Then I can reference using a standard open statement such as open FsXUnit.Extensions. Then the should will be a partially applied function that will tie the functions together. Now we can start throwing our first functions at it. I'll mark each by section giving you a quick sample of each.
Equality
let equal expected actual =
Assert.Equal(expected, actual)
let not_equal expected actual =
Assert.NotEqual(expected, actual)
[<Fact>]
let equal_with_equal_values_should_be_equal() =
"foo" |> should equal "foo"
[<Fact>]
let not_equal_with_inequal_values_should_not_be_equal() =
"foo" |> should not_equal "bar"
Contains
let have expected actual =
Assert.Contains(expected, (actual :> seq<_>))
[<Fact>]
let sequence_should_have_2() =
[1..10] |> should have 2
Exception Handling
let throw_exception<'a when 'a :> exn> actual =
Assert.Throws<'a>(Assert.ThrowsDelegate(actual))
let throws_exception() : unit =
raise(System.ArgumentException "Bad things")
[<Fact>]
let function_should_throw_exception () =
throws_exception |> should (throw_exception<System.ArgumentException>)
This by no means is a complete list of things that you can do with this, but you get the idea.
Where To Go?
As I stated before, I think Scala Specs is an interesting framework for Behavior Driven Development (BDD), and with some of these techniques discussed, it could be rather possible to make a rather expressive framework. It's all about the time for tinkering at this point I suppose. But with all this tinkering I've been doing, I have to wonder to myself, whether we need another testing DSL, or should we more focus on a general purpose language that is best suited for testing, call it Test#, or Fact#, just to humor Brad Wilson... Is that where we should be headed?
-
Thinking in Concurrently in .NET
In recent posts, you've found that I've been harping on immutability and side effect free functions. There is a general theme emerging from this and some real reasons why I'm pointing it out. One of the things that I'm interested in is concurrent programming on the .NET platform for messaging applications. As we see more and more cores and processors available to us, we need to be cognizant of this fact as we're designing and writing our applications. Most programs we write today are pretty linear in nature, except for say forms applications which use background worker threads to not freeze the user interface. But for the most part, we're not taking full advantage of the CPU and its cycles. We need not only a better way to handle concurrency, but a better way to describe them as well. This is where Pi-calculus comes into the picture... But before we get down that beaten path, let's look at a few options that I chose. Not that these aren't all of them, just a select few I chose to analyze.
Erlang in .NET?
For many people, Erlang is considered to be one of the more interesting languages to come out of the concurrent programming field. This language has received little attention until now when we've hit that slowdown of scaling our processor speed and instead coming into multi-core/multi-processor environments. What's interesting about Erlang is that it's a functional language, much like F#, Haskell, OCaml, etc. But what makes it intriguing as well is that it's not a static typed language like the others, and instead dynamic. Erlang was designed to support distributed, fault-tolerant, non-stop real-time applications. Written by Ericsson in the 1980s, it has been the mainstay of telephone switches ever since. If you're interested in listening to more about it, check out Joe Armstrong's appearance on Software Engineering Radio Episode 89 "Joe Armstrong on Erlang". If you want to dig deeper into Erlang, check out the book "Programming Erlang: Software for a Concurrent World" also by Joe Armstrong, and available on Amazon.
How does that lead us to .NET? Well, it's interesting that someone thought of trying to port the language to .NET on a project called Erlang.NET. This project didn't get too far as I can tell, and for obvious impedance mismatch reasons. First off, there is a bit of a disconnect between .NET processes and Erlang processes and how he wants to tackle them. Erlang processes are cheap to create and tear down, whereas .NET ones tend to be a bit heavy. Also the Garbage Collection runs a bit differently instead of a per process approach, the CLR takes a generational approach. And another thing is that Erlang is a dynamic language running on its own VM, so it would probably sit on top of the DLR in the .NET space. Not saying it's an impossible task, but improbable the way he stated.
Instead, maybe the approach to take with an Erlang-like implementation is to create separate AppDomains since they are relatively cheap to create. This will allow for process isolation and messaging constructs to fit rather nicely. Instead, we get rid of the impedance mismatch by mapping an Erlang process to an AppDomain. Then you can tear down the AppDomain after you are finished or you could restart them in case of a recovery scenario. These are some of the ideas if you truly want to dig any further into the subject. I'll probably cover this in another post later.
So, where does that leave us with Erlang itself? Well, we have the option of integrating Erlang and .NET together through OTP.NET. The original article from where the idea came from is from the ServerSide called "Integrating Java and Erlang". This allows for the use of Erlang to do the computation on the server in a manner that best fits the Erlang style. I find it's a pretty interesting article and maybe when I have a spare second, I'll check it out a bit more. But, in terms of a full port to .NET? Well, I think .NET languages have some lessons to learn from Erlang, as it tackled concurrent programming as the first topic instead of most imperative languages bolting it on after the fact.
MPI.NET
The Message Passing Interface (MPI) approach has been an interesting way of solving mass concurrency for applications. This involves using a standard protocol for passing messages from node to node through the system by the way of a compute cluster. In the Windows world, we have Windows Compute Cluster Server (WCCS) that handles this need. CCS is available now in two separate SKUs, CCS 2003 and CCS 2008 for Server 2008. The Server 2008 CCS is available in CTP on the Microsoft Connect website. See here for more information. You mainly find High Performance Computing with MPI in the automotive, financial, scientific and academic communities where they have racks upon racks of machines.
Behind the scenes, Microsoft implemented the MPICH2 version of the MPI specification. This was then made available to C programmers and is fairly low level. Unfortunately, that leaves most C++ and .NET programmers out in the cold when it comes to taking advantage. Sure, C++ could use the standard libraries, but instead, the Boost libraries were created to support MPI in a way that C++ could really take advantage of.
After this approach was taken, a similar approach was taken for the .NET platform with MPI.NET. The University of Indiana produced a .NET version which looked very similar to the Boost MPI approach but with .NET classes. This allows us to program in any .NET language now against the Windows CCS to take advantage of the massive scalability and scheduling services offered in the SKU. At the end of the day, it's just a thin wrapper over P/Invoking msmpi.dll with generics thrown in as well. Still, it's a nice implementation.
And since it was written for .NET, I can for example do a simple hello world application in F# to take advantage of the MPI. The value being is that most algorithms and heavy lifting you would be doing through there would probably be functional anyways. So, I can use F# to specify more succinctly what types of actions and what data I need. Here is a simple example:
#light
#R "D:\Program Files\MPI.NET\Lib\MPI.dll"
open MPI
let main (args:string[]) =
using(new Environment(ref args))( fun e ->
let commRank = Communicator.world.Rank
let commSize = Communicator.world.Size
match commRank with
| 0 ->
let intValue = ref 0
[1 .. (commSize - 1)] |> List.iter (fun i ->
Communicator.world.Receive(Communicator.anySource, Communicator.anyTag, intValue)
System.Console.WriteLine("Hello from node {0} out of {1}", !intValue, commSize))
| _ -> Communicator.world.Send(commRank,0, 0)
)
main(System.Environment.GetCommandLineArgs())
I'll go into more detail in the future as to what this means and why, but just to whet your appetite about what you can do in this is pretty powerful.
F# to the Rescue with Workflows?
Another topic for discussion is for asynchronous workflows. This is another topic in which F# excels as a language. Async<'a> values are really a way of writing continuation passing explicitly. I'll be covering this more in a subsequent post shortly, but in the mean time, there is good information from Don Syme here and Robert Pickering here.
Below is a quick example of an asynchronous workflow which fetches the HTML from each of the given web sites. I can then run each in parallel and get the results rather easily. What I'll do below is a quick retrieval of HTML by calling the Async methods. Note that these methods don't exactly exist, but F# through its magic, creates that for you.
#light
open System.IO
open System.Net
open Microsoft.FSharp.Control.CommonExtensions
let fetchAsync (url:string) =
async { let request = WebRequest.Create(url)
let! response = request.GetResponseAsync()
let stream = response.GetResponseStream()
let reader = new StreamReader(stream)
let! html = reader.ReadToEndAsync()
return html
}
let urls = ["http://codebetter.com/"; "http://microsoft.com"]
let htmls = Async.Run(Async.Parallel [for url in urls -> fetchAsync url])
print_any htmls
So, as you can see, it's a pretty powerful mechanism for retrieving data asynchronously and then I can run each of these in parallel with parameterized data.
Parallel Extensions for .NET
Another approach I've been looking at is the Parallel Extensions for .NET. The current available version is for the December CTP and is available here. You can read more about it from two MSDN Magazine articles:
What I find interesting is Parallel LINQ or PLINQ for short. The Task Parallel library doesn't interest me as much. LINQ in general is interesting to a functional programmer in that it's a lazy loaded function. The actual execution of your LINQ task is delayed until the first yield in GetEnumerator() has been called. That's definitely taking some lessons from the functional world and pretty powerful. And add on top of that the ability to parallelize your heavy algorithms is a pretty powerful concept. I hope this definitely moves forward.
Conclusion
As you can see, I briefly gave an introduction to each of these following areas that I hope to dive into a bit more in the coming weeks and months. I've only scratched the surface on each and each tackle the concurrency problems in slightly different ways and each has its own use. But I hope I whetted your appetite to look at some of these solutions today.
-
DC ALT.NET Meeting 4/23/2008 - Jay Flowers and CI Factory
Now that we've somewhat recovered from ALT.NET Open Spaces, Seattle, it's time for another DC ALT.NET meeting. I'm currently finishing up my wrapups for Seattle still and I'm sure I have months worth of material from there. Anyhow, this time Jay Flowers will be talking to us about Continuous Integration and CI Factory which was postponed from last month due to schedule conflicts. As always we have the first hour or whenever the conversation ends for our main topic and the rest is Open Spaces. Food will be served as well.
Below are the details for the meeting:
Time:
4/23/2008 - 7PM-9PM
Location:
2201 Wilson Blvd
Arlington, VA 22201
Parking/Metro:
Best parking on N. Veitch St
Courthouse Metro the best bet
As always you can find out more by joining the mailing list here. Hope to see a great crowd there and to continue some of the great discussions that were held in Seattle. Until next time...
-
CMAP Code Camp Wrap Up - Dependency Injection and IoC Containers
I really enjoyed speaking at this past weekend's CMAP Code Camp. I hope you all enjoyed my presentation on "Loosen Your Dependencies with Dependency Injection and Inversion of Control Containers". It was a great discussion to have with everyone and I like to learn there as much as I teach.
I also enjoyed teaming up with Scott Allen on his "A Gentle Introduction to Mocking" where we talked about mocks versus stubs, test pattens and mock frameworks such as Rhino Mocks and Moq. Hopefully we'll be doing some more ping-pong sessions in the future.
Once again, I'd like to plug my DC ALT.NET group that I run. Our next meeting is scheduled for April 23rd and the topic will be Continuous Integration with Jay Flowers. We'll announce the location shortly for our group. You can go ahead and sign up for our mailing list here.
Anyhow, here are some resources that can help point you in the right direction. This includes articles, blogs and such that I find will be useful in your journey to better understand these things:
- Articles
- Blogs
- Books
If you note, I my code uses the following products in order to get it to run:
- ASP.NET MVC Preview 2
- xUnit.net
- Castle Windsor
- StructureMap
- Unity Application Block
- Unity Community Contributions
-
xUnit.net RC3 Just Released
Well, Brad Wilson and Jim Newkirk must really be busy lately. After I talked about the release of xUnit.net RC2, just today, Brad announced the release of RC3. As always, you can find the latest bits here. This fixes a number of bugs and adds CruiseControl.NET and ASP.NET MVC Preview 2 support as well in addition to the Resharper 3.1 and TestDriven.NET support. For more information about it, check out Brad's post here. More or less, they are feature complete for version 1.0 and the only that I think really is needed at this point is a decent GUI runner and that's well acknowledged as something they are working on. Visual Studio integration would be nice as well...
If you were in attendance at last night's RockNUG appearance, all tests for my demos were using xUnit.net, so I am actively using it right now and will be for my CMAP Code Camp appearance this weekend. However, I did not show the GUI runner because, well, it's not there yet, and instead, the console runner works just fine, thank you. So, go ahead and pick up the latest bits and give the team feedback!
For my other posts in this series, check them out here:
One last note regarding Brad, he was recently interviewed by Scott Swigart and Sean Campbell over at How Software Is Built and gives some interesting insights in the open source world inside and outside Microsoft and his contributions to it. Very good interview and well worth the time to read.
-
RockNUG IoC Container Presentation Wrapup
I want to thank the fine folks at the Rockville .NET Users Group (RockNUG) and Dean Fiala for giving me the opportunity to speak last night. It was a record crowd last night, so I'm glad that people were interested in Loose Coupling, Design Patterns, Test Driven Development, Behavior Driven Development and Inversion of Control containers. I hope everyone got some good information, and if not interested in using containers, design patterns and such, at least know they exist and have their use. Based on the feedback I've already received, it was warming and why I like presenting at user groups, so that both of us can learn.
Once again, I'd like to plug my DC ALT.NET group that I run. Our next meeting is scheduled for April 23rd and the topic will be Continuous Integration with Jay Flowers. We'll announce the location shortly for our group. You can go ahead and sign up for our mailing list here.
Anyhow, here are some resources that can help point you in the right direction. This includes articles, blogs and such that I find will be useful in your journey to better understand these things:
- Articles
- Blogs
- Books
-
Unity 1.0 Released into the Wild
As Chris Tavares mentioned in his blog, Unity 1.0 has been released a couple of days earlier than the April 7th release date mentioned by Grigori Melnik earlier. Scott Densmore also announced this as well as working on porting the inteception from ObjectBuilder2 which I talked about earlier in some of my Unity and IoC containers posts. Looking forward to that post as we've shared some emails on the subject.
Would You Like To Know More?
For those looking for my posts on the matter, I've covered it quite extensively with the comparison to other IoC containers as well as IoC containers in general:
- IoC and Unity - Configuration Changes for the Better
Covers the latest configuration changes to allow for better constructor injection
- IoC and Unity - The Basics and Inteception
Covers the basics of an IoC container and inteception techniques
- IoC Container, Unity and Breaking Changes Galore
Covers the breaking changes made from the old Unity drop to the new one
- IoC Containers, Unity and ObjectBuilder2 - The Saga Continues
Managing instances and parameter mapping resolution
- IoC and the Unity Application Block Once Again
Setter Injection versus Constructor Injection and PostSharp4Unity
- IoC and the Unity Application Block - Going Deeper
Constructor Injection and comparing Unity with Castle Windsor
- IoC and the Unity Application Block
Covering ObjectBuilder and Unity Application Block
Anyhow, if interesting in more Unity content, check out David Hayden's Unity IoC and MVC screencast as well as others on the subject here.
Speaking of which, I'm going to be a busy man indeed with my upcoming speaking schedule on IoC containers, not necessarily Unity in particular, but all of them, the background and decoupling your applications. Here is the complete schedule for this month:
- RockNUG - April 9th
- CMAP Code Camp - April 12th
- CMAP Architecture SIG - April 15th
- IoC and Unity - Configuration Changes for the Better
-
Relooking at xUnit.net RC2
using MVC_APPLICATION_NAMESPACE.Controllers; // This using directive needs to point to the namespace of your MVC project
-
xUnit.net RC2 Released
public class DatabaseFixture : IDisposable