Glavs Blog

Owin, Katana and getting started

Glav — Thu, 04 Apr 2013 22:19:22 GMT

Introduction

This article describes an emerging open source specification, referred to as Owin – An Open Web Interface for .Net, what it is, and how this might be beneficial to the .Net technology stack. It will also provide a brief, concise look at how to get started with Owin and related technologies. In my initial investigations with Owin and how to play with it, I found lots of conflicting documentation and unclear ways of how to make it work and why. This article is an attempt to better articulate those steps so that you may save yourself lots of time and come to a clearer understanding in a much shorter time than I did.

Note: The code for this article can be download from here.

First, What is it?

Just in case you are not aware, a community driven project referred to as Owin is really gaining traction. Owin is a simple specification that describes how components in a HTTP pipeline should communicate. The details of what is in the communication between components is specific to each component, however there are some common elements to each. Owin itself is not a technology, just a specification.

I am going to gloss over many details here in order to remain concise, but at its very core, Owin describes 1 main component which is the following interface:

Func<IDictionary<string, object>,Task>

This is a function that accepts a simple dictionary of objects, keyed by a string identifier. The function itself returns a task. The object in the dictionary in this instance will vary depending on what the key is referring to.

More often, you will see it referenced like this:

using AppFunc = Func<IDictionary<string, object>,Task>

An actual implementation might look like this:

public Task Invoke(IDictionary<string, object> environment) 
{ 
   var someObject = environment[“some.key”] as SomeObject; 
   // etc… 
}

This is essentially how the “environment” or information about the HTTP context is passed around. Looking at the environment argument of this method, you could interrogate it as follows:

var httpRequestPath = environment[“owin.RequestPath”] as string; 
Console.WriteLine(“Your Path is: [{0}]”,httpRequestPath);

if a HttpRequest was being made to ‘http://localhost:8080/Content/Main.css” then the output would be:

Your Path is [/Content/Main.css]

In addition, while not part of the spec, the IAppBuilder interface is also core to the functioning of an Owin module (or ‘middleware’ in Owin speak):

public interface IAppBuilder
{
IDictionary<string, object> Properties { get; }

    object Build(Type returnType);
    IAppBuilder New();
    IAppBuilder Use(object middleware, params object[] args);
}

The IAppBuilder interface acts as the ‘glue’ or host to bring any registered Owin compatible libraries/modules together.

So what? I can do that now without Owin.

Okay so it may not look ground breaking, but this is the core of Owin and key to understanding what it offers. Since we have a basic understanding of the how, I am going to jump to what it currently offers as a result of supporting this mechanism. With support of this simple mechanism, I can now write isolated components that deal with specific parts of functionality related to Http requests. I can then chain them together to build capabilities of my Http server. Internet Information Server does not need to get a look in. You can literally chain together Owin components to form a pipeline of only the necessary features you want. In addition, the components I write do not have to have specific references to any particular pipeline component, and yet can still take advantage of any Owin compatible component.

Basically you can build or use a custom host, then you can insert whatever custom modules into the Http request processing pipeline. Owin provides the specification for writing those modules that make it easy to insert into the chain.

Internet Information Services comes with a plethora of pipeline and request handlers, management functionality, knobs, dials and features that make a comprehensive web server. However many people want a lean, cut down web server with only the features they want. This provides a much leaner pipeline that can provide great performance, in addition to only providing the feature set you want, meaning a smaller surface area which can be beneficial from a complexity and sometimes a security perspective.

Quit yer jibba jabba fool, show me some code

Yes yes, I will, but first, a diagram illustrating the previous paragraph. As already mentioned, Owin is just a specification of what interface to expose and how to work with that interface.In order for modules written to the Owin spec to be functional, they must exist in a host. So at a conceptual level, Owin compared to Internet Information Server looks like this:

Since we have an empty Owin host, we can programmatically add Owin compatible modules based on only what our needs are as illustrated below:

The diagram above shows an Owin host with 3 modules registered to be used, Static files, Authentication and SignalR.

(Note: SignalR is a popular real-time web messaging/connection library and has been written as an Owin compatible module)

Are we there yet?

Ok, so lets finally write an Owin compatible module or library.

Within Visual Studio, Create a new blank solution.
Add a new class library project to it.
Install the Owin nuget package using:
Install-Package Owin
(via the package manager console in Visual Studio or right click on the project in Visual Studio and select Manage Nuget Packages)
Add a new class, lets call it TestLogger and paste in the following code:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using AppFunc = Func<IDictionary<string, object>, Task>;
    public class TestLogger
    {
        private readonly AppFunc _next;
        public TestLogger(AppFunc next)
        {
            if (next == null)
            {
                throw new ArgumentNullException("next");
            }
            _next = next;
        }
        public Task Invoke(IDictionary<string, object> environment)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Hitting TestLogger, path: {0}", environment["owin.RequestPath"]));
            return _next(environment);
        }
    }

That is actually all you need to write an Owin compatible library. Strictly speaking, you don’t need the reference to the Owin nuget package but we will make use of that in the next part. All this module does is write out the contents of the request path to the trace output for each request that comes in. All we need to do is examine the environment that is provided to us through the environment variable.

However, we have a module but nothing to host it in and no way to tell it to be included as part of a web server pipeline. We need an Owin capable host that loads our module and houses the running code. We also need to programmatically configure that host to include our module. Much like Internet Information Server acts as the host for most web apps in .Net..

So, we are going to add a web project to our solution which will act as the entry point, allowing us to configure a module for use in the web request pipeline.

Within Visual Studio, add a new empty MVC project (I used MVC4) to your current solution.
You need to reference at least the Owin assemblies. In addition, we are going to reference the Owin HttpListener written by Microsoft to act as the actual Http server in our Owin pipeline. To do this:
Install the Owin nuget package and the Microsoft.Owin.HttpListener package using:
Install-Package Owin
Install-Package Microsoft.Owin.Host.HttpListener -Pre
(via the package manager console in Visual Studio or right click on the project in Visual Studio and select Manage Nuget Packages. Note that the Microsoft.Owin.HttpListener package is pre-release so we need include the –pre option.)
After installing those packages, you should see the appropriate references in your project as shown in the screen grab below:
Add a new class to this MVC project called ‘Startup’
Paste in the following code:

public class Startup
{
    // Invoked once at startup to configure your application.
    public void Configuration(IAppBuilder builder)
    {
        builder.Use(typeof(TestLogger));
    }
}

Here you can see we are using an instance of IAppBuilder passed into our configuration method to tell the host to ‘Use’ our custom pipeline component, often referred to as ‘Middleware’ in Owin speak.

Owin components use conventions for easy configuration, so when the Owin host starts up and loads our assemblies, it looks for the Startup class, and also a ‘Configuration’ method which accepts an IAppBuilder instance, and calls that, providing the instance.

The final piece of the puzzle

Ok, we have our custom pipeline component, our middleware, and we have an entry point where we can build or configure our pipeline. We need a host process to load these assemblies and invoke them with the requisite information.

This is where Project Katana comes in. Katana is a Microsoft written generic Owin host. You can go grab the source of this project, compile and run it, but it is easier to just install it so you can call it from the command line. By far, the easiest way to do this is to install a tool called ‘Chocolatey’ which is like a Nuget package installer but for windows binaries.

(Side Note: Chocolately is an awesome tool. If you get nothing else from this post, just use Chocolately to install some software and keep it up to date. It is easy to use and has a heap of applications supported.)

To install Chocolotey, run the following from a command line (literally copy and paste into a command line/console window):
@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

Once its installed, from the command line, simply type:
cinst Katana –pre

And Katana will be magically installed. Now opening up a command line/console window and typing ‘Katana’ will load and run the Katana application, which is our Microsoft written Owin host. Nothing will get loaded because we haven’t specified anything, but that is what is next.

We have written our custom Owin component, we have created our web application entry point used to configure our Owin pipeline and we have installed our host application, Katana, to glue it all together. So lets see in action.

First we need to tell our web project to NOT automatically use the current page when starting up. Rather, we want it to startup the Katana host, and tell Katana to use our assemblies. To do this:

Open up the properties of the web project in your solution and select the ‘Web’ section. You should see a screen similar to the one shown below:.

For clarification, the things you need to enter are:

Ensure the ‘Start External Program’ is selected and enter the full path to the Katana executable. I installed to the default location using Chocolately so mine is set at:

C:\Chocolatey\lib\Katana.0.20-alpha-20220-88\tools\Katana.exe

Set the command line arguments to:

-p8080 –v

This represent the port number of 8080 and enabling verbose mode

Enter the working directory where your web project is located. In my case it was:

D:\Glav.Owin.Middleware\Glav.Owin.Consumer.Test.BasicWebConsumer

Now hit F5 to run the app. You should see a console application launched which looks similar to the one below:

Now remember, the only component we have injected into the Owin pipeline is our own custom logging component. Since we have included the Microsoft.Owin.Host.HttpListener package into our project, that assembly also gets included in the output. When Katana executes, if no specific Http server is defined, it will automatically look for the Microsoft.Owin.Host.HttpListener assembly and use that.

Again, we only have our logging component installed, so if we load a browser, a navigate to http://localhost:8080 , we wont see a web page but the console should show the logging output of our custom component as in the screen shot below:

All that work, for that?

Yes, but now we have everything in place to easily add other components in the pipeline. In addition, now you should have a reasonable understanding of the various pieces at play. Remember, we have created a custom web server that only does exactly what we want it to.So conceptually, what we have created looks roughly like the following:

Now lets create another component but I don’t want to create another project for that.

Add the following method block to your ‘Startup’ class in the web project in your solution:

// Invoked once per request.
public Task Invoke(IDictionary<string, object> environment)
{
   var responseBytes = System.Text.ASCIIEncoding.UTF8.GetBytes(
            string.Format("Serviced request on {0} at {1}",DateTime.Now.ToLongDateString(), DateTime.Now.ToLongTimeString()));
    Stream responseStream = (Stream)environment["owin.ResponseBody"];
    IDictionary<string, string[]> responseHeaders =
        (IDictionary<string, string[]>)environment["owin.ResponseHeaders"];
    responseHeaders["Content-Length"] = new string[] { responseBytes.Length.ToString(CultureInfo.InvariantCulture) };
    responseHeaders["Content-Type"] = new string[] { "text/plain" };
    return responseStream.WriteAsync(responseBytes, 0, responseBytes.Length);
}

Then add this line of code as the last line in the ‘Configuration’ method within the ‘Startup’ class.

builder.Use(new Func<AppFunc, AppFunc>(ignoredNextApp => (AppFunc)Invoke));

Now, hit F5 to run your app, and browser to http://localhost:8080

You should see a response in the browser similar to the screen grab shown below:

That code we added in the Invoke method, did the following:

Created a simple text response as a byte array.
Grabbed the response body stream and the response headers object from the environment variable using standard Owin defined key names.
Set the response headers according to the response itself.
Wrote the bytes to the response stream.

In order to use the code in the Invoke method, we needed to tell our pipeline to use it. This was achieved in the Configuration method via the line:

builder.Use(new Func<AppFunc, AppFunc>(ignoredNextApp => (AppFunc)Invoke));

This simply injected the delegate for our Invoke function into the pipeline. So now our comceptual pipeline looks like this:

All without any real dependencies and in a relatively small amount of code.

And you made it

After all that, you can finally see how composable and modular the Owin concepts and architecture are. There are quite a few Owin compatible middleware modules out there such as SignalR, FubuMVC, NancyFx to name a few. These can be added to your pipeline and you then have the power that each brings at your disposal.

This is just the beginning. Imagine being able to easily build a custom web server with only the capabilities you need. An open, composable and modular world with no lock in. Sounds good to me. Hopefully the Owin concepts and goals are a little clearer now.

Remember, the code for this article can be downloaded from here.

Debugging Pain–End to End

Glav — Wed, 20 Feb 2013 21:18:47 GMT

We had an issue recently that caused us some time and quite a lot of head scratching. We had made some relatively minor changes to our product and performed a release into staging for testing. We released our main web application as well as our custom built support tool (also a web app).

After a little bit of testing from our QA team, a few bugs were uncovered. One where a response to a cancel action seemingly was not actioned, and an issue where a timeout occurred on a few requests. Nothing too huge and certainly seemed fixable.

Off to work

The timeouts “seemed” to be data specific and possibly because of our 3rd party web service call being made. It seemed to be only occurring in our support tool, the main web app was not affected. Since the main web app is the priority, I looked at the “cancel” issue not working. It seemed that the cancel request was being made to our server (via an ajax call) but never returning from said call. This looked very similar to our issue with the support tool.

A little further investigation showed that both the support tool and our main web app were issuing ajax requests to a few action methods (we use ASP.Net MVC 4.5) and never returning. Ever. I tried recycling the application pool in IIS. This worked for a brief period, then requests to a few particular requests to action methods were not returning. Web pages from other action methods and even other ajax requests were working fine so at least we knew what surface area we had to look at.

Looking at the requests via Chrome, we could see the request in a constant pending state, never satisfied. We began looking at the server. We instituted some page tracing, looked at event logs and also looked at the Internet Information Server logs. Nothing. Nada. We could see the successful requests come in, but these pending requests were not logged. Fiddler showed they were definitely outgoing, but the web server showed nothing.

Using the Internet Information Service management console, we looked at the current worker processes which is available when clicking on the root node within the IIS Management console.

We could see our application pool and right clicking on this allowed us to view current requests, and there they were, all waiting to be executed, and waiting…..and waiting.

What’s the hold up?

So what was causing our requests to get backlogged? We tried going into the code and removing calls to external services and trying to isolate the problem areas. All of this was not reproducible locally, nor in any other environment. Eventually trying to isolate the cause led us to removing everything from the controller actions apart from simple Thread.Sleep.While this worked and the problem did not present, we were in no way closer as it was still in potentially any number of code paths.

Take a dump

A colleague suggested using DebugDiag (a free diagnostic tool from Microsoft) to look at memory dumps of the process. So that is what we did.

Using DebugDiag, we extracted a memory dump of the process. DebugDiag has some really nice features, one of which is to execute predefined scripts in an attempt to diagnose any issues and present a summary of what was found and also has a nice wizard based set of of steps to get you up and running quickly.

We chose to monitor for performance:

and also for HTTP Response time:

We then added the specific URL’s we were monitoring. We also chose what kind of dumps to take, in this case web application pools:

We decided on the time frequency (we chose every 10 seconds) and a maximum of 10 Full dumps:

After that, we set the dump path, named and activated the rule, and we good to go. With the requests already built up in the queue and issuing some more ‘pending’ requests, we could see some memory dumps being taken.

A cool feature of DebugDiag is the prebuilt scripts to analyze your memory dumps (available on the advanced tab):

We initially chose performance, but didn’t glean much information from that. We then chose the “Crash/Hang Analyzers” which produced great summary of all the threads in the pool. It was apparent that almost every thread was waiting on a .Net Lock. We couldn’t get much more than that though.

WinDbg to the rescue

So I copied the memory dump locally and use WinDbg to examine the memory process. I loaded in the SOS extension so I could use the managed memory extensions.

(Side Note: I almost always have issues with incorrect format –32 or 64 bit- and SOS versions when doing this so it usually takes a bit of frigging around before I get the right combination)

I looked at the threads via:

!threads

Sure enough, there were heaps of them in there. Using Dump object (!do) and trying to poke around didn’t reveal too much except a lot of thread locks. So I used the !syncblk command to look at blocked and locking threads:

Here we had some real indication of what was going on. You can see we have 285 locks via the Apache.NMS functions. Our application uses ActiveMQ (which is awesome btw) for our message queuing and we use the Apache.NMS library as the C# interface. In addition, the code paths being executed utilise extensive diagnostic information among all code paths. This diagnostic information is posted to the Message Queue for logging.

A quick test verified this. We commented out all calls to the queue within the queue manager interface (so it effectively did nothing). Put this code on staging and all was working without a hitch.

So we had our culprit, but not the root cause.

We used the admin tool for ActiveMQ to look at the queues themselves.Some queued messages, but not subscribers even though our subscription service was running. Restarted the service, nothing. No subscribers. Using the admin tool we purged all messages. Restarted the service. Nothing. Refreshed the admin tool, the purged messages re-appeared!

We tried deleting the offending queues.Normally, this operation is sub-second. In this case, it took 20+ seconds before we tried again. Something was amiss. We tried creating a new queue from the admin tool.Again, an abnormally long time, but this time it did it. We could then delete queues without issue. Restarted the service and viola, queues subscribed to. We re-instituted the commented code and now all working fine.

So what really happened?

A short time ago, we had run out of space on our staging server. No big deal, freed up the space promptly, all seemingly good. it was during this time that I believe that our message queue repository, which persists messages to disk, got corrupted and this started the issue occurring. What is apparent is that we need to release resources more aggressively so that this issue would not be so detrimental to the rest of the application function.

So there you have it. All in all that took about 2-3 days.

CacheAdapter minor update to version 2.5.1

Glav — Wed, 23 Jan 2013 03:42:57 GMT

For information on previous releases, please see here.

My cache adapter library has been updated to version 2.5.1. This one contains only minor changes which are:

Use of the Windows Azure Nuget package instead of the assemblies directly referenced.
Use of the ‘Put’ verb instead of ‘Add’ when using AppFabric to prevent errors when calling ‘Add’ more than once.
Minor updates to the example code to make it clearer that the code is indeed working as expected.

Thanks to contributor Darren Boon for forking and providing the Azure updates for this release.

Writing an ASP.Net Web based TFS Client

Glav — Tue, 27 Nov 2012 01:53:17 GMT

So one of the things I needed to do was write an ASP.Net MVC based application for our senior execs to manage a set of arbitrary attributes against stories, bugs etc to be able to attribute whether the item was related to Research and Development, and if so, what kind.

We are using TFS Azure and don’t have the option of custom templates. I have decided on using a string based field within the template that is not very visible and which we don’t use to write a small set of custom which will determine the research and development association.

However, this string munging on the field is not very user friendly so we need a simple tool that can display attributes against items in a simple dropdown list or something similar.

Enter a custom web app that accesses our TFS items in Azure (Note: We are also using Visual Studio 2012)

Now TFS Azure uses your Live ID and it is not really possible to easily do this in a server based app where no interaction is available. Even if you capture the Live ID credentials yourself and try to submit them to TFS Azure, it wont work.

Bottom line is that it is not straightforward nor obvious what you have to do. In fact, it is a real pain to find and there are some answers out there which don’t appear to be answers at all given they didn’t work in my scenario.

So for anyone else who wants to do this, here is a simple breakdown on what you have to do:

Go here and get the “TFS Service Credential Viewer”. Install it, run it and connect to your TFS instance in azure and create a service account. Note the username and password exactly as it presents it to you. This is the magic identity that will allow unattended, programmatic access.
In your MVC app, reference the following assemblies from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v2.0”:
- Microsoft.TeamFoundation.Client.dll
- Microsoft.TeamFoundation.Common.dll
- Microsoft.TeamFoundation.VersionControl.Client.dll
- Microsoft.TeamFoundation.VersionControl.Common.dll
- Microsoft.TeamFoundation.WorkItemTracking.Client.DataStoreLoader.dll
- Microsoft.TeamFoundation.WorkItemTracking.Client.dll
- Microsoft.TeamFoundation.WorkItemTracking.Common.dll
If hosting this in Internet Information Server, for the application pool this app runs under, you will need to enable 32 Bit support.

You also have to allow the TFS client assemblies to store a cache of files on your system. If you don’t do this, you will authenticate fine, but then get an exception saying that it is unable to access the cache at some directory path when you query work items. You can set this up by adding the following to your web.config, in the <appSettings> element as shown below:

<appSettings> <!-- Add reference to TFS Client Cache -->
   <add key="WorkItemTrackingCacheRoot" value="C:\windows\temp" />
</appSettings>

With all that in place, you can write the following code:

var token = new Microsoft.TeamFoundation.Client.SimpleWebTokenCredential("{you-service-account-name", "{your-service-acct-password}");
var clientCreds = new Microsoft.TeamFoundation.Client.TfsClientCredentials(token);
var currentCollection = new TfsTeamProjectCollection(new Uri(“https://{yourdomain}.visualstudio.com/defaultcollection”), clientCreds);

TfsConfigurationServercurrentCollection.EnsureAuthenticated();

In the above code, not the URL contains the “defaultcollection” at the end of the URL. Obviously replace {yourdomain} with whatever is defined for your TFS in Azure instance. In addition, make sure the service user account and password that was generated in the first step is substituted in here.

Note: If something is not right, the “EnsureAuthenticated()” call will throw an exception with the message being you are not authorised. If you forget the “defaultcollection” on the URL, it will still fail but with a message saying you are not authorised. That is, a similar but different exception message.

And that is it. You can then query the collection using something like:

var service = currentCollection.GetService<WorkItemStore>();

var proj = service.Projects[0];
var allQueries = proj.StoredQueries;
for (int qcnt = 0; qcnt < allQueries.Count; qcnt++)
{
var query = allQueries[qcnt];
var queryDesc = string.format(“Query found named: {0}”,query.Name);
}

You get the idea.

If you search around, you will find references to the ServiceIdentityCredentialProvider which is referenced in this article. I had no luck with this method and it all looked too hard since it required an extra KB article and other magic sauce.

So I hope that helps. This article certainly would have helped me save a boat load of time and frustration.

CacheAdapter 2.5–Memcached revised

Glav — Wed, 16 May 2012 23:14:51 GMT

Note: For more information around the CacheAdapter, see my previous posts here, here, here and here

You may have noticed a number of updates to the CacheAdapter package on nuget as of late. These are all related to performance and stability improvements for the memcached component of the library.

However it became apparent that I needed more performance for the memcached support to make the library truly useful and perform as best as can be expected.

I started playing with optimising serialisation processes and really optimising the socket connections used to communicate with the memcached instance. As anybody who has worked with sockets before may tell you, you quickly start to look at pooling your connections to ensure you do not have to go about recreating a connection every time you want to communicate, especially if we are storing or retrieving items in a cache as this can be very frequent.

So I started implementing a pooling mechanism to increase the performance. I got to improving it to a reasonable extent, then found I would hit a hurdle where performance could be increased further but it was becoming harder and more complex to retain stability. I became lost in trying to fix the problem when a solution had already been in place for a long time.

It was about then I decided it best to simply take a dependency on the most excellent Enyim memcached client. It is fast. Really fast, and blew away everything I had done in terms of performance. So that is the essence of this update. The memcached support in the cache adapter comes courtesy of Enyim. Enjoy the extra speed that comes with it.

Note: There are no other updates to any of the other caching implementations.

CacheAdapter 2.4 – Bug fixes and minor functional update

Glav — Wed, 28 Mar 2012 20:22:54 GMT

Note: If you are unfamiliar with the CacheAdapter library and what it does, you can read all about its awesome ability to utilise memory, Asp.Net Web, Windows Azure AppFabric and memcached caching implementations via a single unified, simple to use API from here and here..

The CacheAdapter library is receiving an update to version 2.4 and is currently available on Nuget here.

Update: The CacheAdapter has actualy just had a minor revision to 2.4.1. This significantly increases the performance and reliability in memcached scenario under more extreme loads. General to moderate usage wont see any noticeable difference though.

Bugs

This latest version fixes a big that is only present in the memcached implementation and is only seen in rare, intermittent times (making i particularly hard to find). The bug is where a cache node would be removed from the farm when errors in deserialization of cached objects would occur due to serialised data not being read from the stream in entirety.

The code also contains enhancements to better surface serialization exceptions to aid in the debugging process. This is also specifically targeted at the memcached implementation. This is important when moving from something like memory or Asp.Web caching mechanisms to memcached where the serialization rules are not as lenient.

There are a few other minor bug fixes, code cleanup and a little refactoring.

Minor feature addition

In addition to this bug fix, many people have asked for a single setting to either enable or disable the cache.In this version, you can disable the cache by setting the IsCacheEnabled flag to false in the application configuration file. Something like the example below:

  <Glav.CacheAdapter.MainConfig>
    <setting name="CacheToUse" serializeAs="String">
      <value>memcached</value>
    </setting>
    <setting name="DistributedCacheServers" serializeAs="String">
      <value>localhost:11211</value>
    </setting>
    <setting name="IsCacheEnabled" serializeAs="String">
      <value>False</value>
    </setting>
  </Glav.CacheAdapter.MainConfig>

Your reasons to use this feature may vary (perhaps some performance testing or problem diagnosis). At any rate, disabling the cache will cause every attempt to retrieve data from the cache, resulting in a cache miss and returning null. If you are using the ICacheProvider with the delegate/Func<T> syntax to populate the cache, this delegate method will get executed every single time. For example, when the cache is disabled, the following delegate/Func<T> code will be executed every time:

var data1 = cacheProvider.Get<SomeData>("cache-key", DateTime.Now.AddHours(1), () =>
{
    // With the cache disabled, this data access code is executed every attempt to
    // get this data via the CacheProvider.
    var someData = new SomeData() { SomeText = "cache example1", SomeNumber = 1 };
    return someData;
});

One final note: If you access the cache directly via the ICache instance, instead of the higher level ICacheProvider API, you bypass this setting and still access the underlying cache implementation. Only the ICacheProvider instance observes the IsCacheEnabled setting.

Thanks to those individuals who have used this library and provided feedback. Ifyou have any suggestions or ideas, please submit them to the issue register on bitbucket (which is where you can grab all the source code from too)

ASP.NET Web Api–Request/Response/Usage Logging

Glav — Sun, 26 Feb 2012 10:26:00 GMT

For part 1 of this series of blog posts on ASP.Net Web Api – see here.

Introduction

In Part 2 of this blog post series, we deal with the common requirement of logging, or recording the usage of your Web Api. That is, recording when an Api call was made, what information was passed in via the Api call, and what response was issued to the consumer.

My new Api is awesome – but is it being used?

So you have a shiny new Api all built on the latest bits from Microsoft, the ASP.Net Web Api that shipped with the Beta of MVC 4. A common need for a lot of Api’s is to log usage of each call, to record what came in as part of the request, to also record what response was sent to the consumer. This kind of information is really handy for debugging.Not just for you, but also for your customers as well. Being able to backtrack over the history of Api calls to determine the full context of some problem for a consumer can save a lot of time and guesswork.

So, how do we do this with the Web Api?

Glad you asked.

Determining what kind of injection point to utilise

We have a few options when it comes to determining where to inject our custom classes/code to best intercept incoming and outgoing data. To log all incoming and outgoing data, the most applicable interception point is a System.Net.Http.DelegatingHandler. These classes are message handlers that apply to all messages for all requests and can be chained together to have multiple handlers registered within the message handling pipeline. For example, in addition to Api usage logging, you may want to provide a generic authentication handler that checks for the presence of some authentication key. I could have chosen to use filters however these are typically more scoped to the action itself. I could potentially use model binders but these are relatively late in the processing cycle and not generic enough (plus it would take some potentially unintuitive code to make it work as we would want)..

Enough with theory, show me some code

So, our initial Api usage logger will inherit from the DelegatingHandler class (in the System.Net.Http namespace) and provide its own implementation.

public class ApiUsageLogger : DelegatingHandler
{
    protected override System.Threading.Tasks.Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
    {
        //your stuff goes here...
    }
}

We only really need to override one method, the ‘SendAsync' method which returns a Task. Task objects play heavily in the new Web Api and allow asynchronous processing to be used as the primary processing paradigm allowing better scalability and processing utilisation.

Since everything is asynchronous via the use of Tasks, we need to ensure we play by the right rules and utilise the asynchronous capabilities of the Task class. A more fleshed out version of our method is shown below:

protected override System.Threading.Tasks.Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
{
    // Extract the request logging information
    var requestLoggingInfo = ExtractLoggingInfoFromRequest(request);

    // Execute the request, this does not block
    var response = base.SendAsync(request, cancellationToken);

    // Log the incoming data to the database
    WriteLoggingInfo(requestLoggingInfo);

    // Once the response is processed asynchronously, log the response data
    // to the database
    response.ContinueWith((responseMsg) =>
    {
        // Extract the response logging info then persist the information
        var responseLoggingInfo = ExtractResponseLoggingInfo(requestLoggingInfo, responseMsg.Result);
        WriteLoggingInfo(responseLoggingInfo);
    });
    return response;
}

To step through the code:

First, we extract relevant information from the request message.
We execute the request asynchronously via the base class SendAsync method passing in what we received.
We log the request information (perhaps to a database). Note that this operation could be performed asynchronously as well.
We then define an anonymous function that the asynchronous request task should execute once it has completed.This is done using the ‘ContinueWith’ construct, and passing in an instance of the response message
We return the ‘response’, which is really just a Task object that will process the request asynchronously and execute our response logging code once the request has completed processing. .

Registering our Handler

We have created our custom DelegatingHandler, so we need to tell the framework about it.

The registration is done at application startup through access to the GlobalConfiguration object provided by the framework. In the global.asax.cs file, you will find the Application_Start method looking something like this:

protected void Application_Start()
{
    RouteTable.Routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
    RouteTable.Routes.IgnoreRoute("{resource}.ico");

    RegisterApis(GlobalConfiguration.Configuration);
}

You can see the GlobalConfiguration.Configuration object (which is of type System.Web.Http.HttpConfiguration) being passed to the RegisterApis method. We then simply register our DelegatingHandler into the list of MessageHandler from the configuration object as in the following code:

public static void RegisterApis(HttpConfiguration config)
{
    config.MessageHandlers.Add(new ApiUsageLogger());
}

Note: If you have dependency injection setup for your Web Api project, you can either pass in the service resolver as a constructor argument when registering this handler (in the global.asax.cs for example ) like:

config.MessageHandlers.Add(new ApiUsageLogger(_myContainer));

or you can use that same resolver to resolve an instance of this DelegatingHandler when adding to the MessageHandler collection as in:

config.MessageHandlers.Add(_myContainer.Resolve<IApiUsageLogger>());

I shall address some of the nuances of DI in a separate post.

And that is it. Well, that is all that is required to register a new handler that will get called for each request coming into your Api. Actually logging the information requires a little more work to ensure we play correctly in this nice asynchronous world of the WebApi.

But wait, there’s more

Playing nice in the asynchronous world of the WebApi requires a little bit of care. For example, to extract out request information, particularly the body of the request message, you might write a method like the following:

private ApiLoggingInfo ExtractLoggingInfoFromRequest(HttpRequestMessage request)
{
    var info = new ApiLoggingInfo();
    info.HttpMethod = request.Method.Method;
    info.UriAccessed = request.RequestUri.AbsoluteUri;
    info.IPAddress = HttpContext.Current != null ? HttpContext.Current.Request.UserHostAddress : "0.0.0.0";

    ExtractMessageHeadersIntoLoggingInfo(info, request.Headers.ToList());
    if (request.Content != null)
    {
        var byteResponse = request.Content.ReadAsByteArrayAsync().Result;
        info.BodyContent = System.Text.UTF8Encoding.UTF8.GetString(byteResponse);
    }
    return info;
}

Notice the line that reads:

var byteResponse = request.Content.ReadAsByteArrayAsync().Result;

Here we are accessing the Result property from an asynchronous task in an attempt to make this code procedural and work in a synchronous manner. It is an easy thing to do, and looks like it makes sense. However, do not do this.

Generally, you should never access the ‘Result’ property of a Task unless you know that the task has completed as this can cause deadlocks in ASP.Net. Yes it is true.Sometimes this code may work but, if you don’t want to waste hours debugging deadlock issues in ASP.Net, I would advise against it.

So how do we know when it is complete? With the ‘ContinueWith’ construct of course. So your code may change to look more like this:

private ApiLoggingInfo ExtractLoggingInfoFromRequest(HttpRequestMessage request)
{
    var info = new ApiLoggingInfo();
    info.HttpMethod = request.Method.Method;
    info.UriAccessed = request.RequestUri.AbsoluteUri;
    info.IPAddress = HttpContext.Current != null ? HttpContext.Current.Request.UserHostAddress : "0.0.0.0";

    ExtractMessageHeadersIntoLoggingInfo(info, request.Headers.ToList());
    if (request.Content != null)
    {
        request.Content.ReadAsByteArrayAsync()
            .ContinueWith((task) =>
                            {
                                info.BodyContent = System.Text.UTF8Encoding.UTF8.GetString(task.Result);
                                return info;

                            });
    }
    return info;
}

In the above example, we can safely use the ‘Result’ property of the task since the method within the ‘ContinueWith’ block is only executed once the initial task is completed.

Wrap up

The above code fragments provide a simple and basic way of registering a custom interception point within the WebApi pipeline to perform a custom task, in this case usage logging.

The next series of posts will look at items such as the use of model binding, security and much more.

MVC4 and Web Api– make an Api the way you always wanted–Part 1

Glav — Sat, 18 Feb 2012 01:33:49 GMT

Update: Part 2 of this series is available here.

ASP.NET MVC is a huge success as framework and just recently, ASP.NET MVC4 Beta was released. While there are many changes in this release, I want to specifically focus on the WebApi portion of this.

In previous flavours of MVC, many people wanted to develop REST Api’s and didn’t really want to use WCF for this purpose. The team at Microsoft was progressing along with a library called WcfWebApi. This library used a very WCF’esque way of defining an Api. This mean defining an interface and decorating the interface with the appropriate WCF attributes to constitute your Api endpoints.

However, a lot of people don’t like to use a WCF style of programming, and are really comfortable in the MVC world. Especially when you can construct similar REST endpoints in MVC with extreme ease. This is exactly what a lot of people who wanted a REST Api did. They simply used ASP.NET MVC to define a route and handled the payload themselves via standard MVC controllers.

What the WcfWebApi did quite well though was things like content negotiation (do you want XML or json?), auto help generation, message/payload inspection, transformation, parameter population and a lot of other things.

Microsoft have recognised all this and decided to mix it all together. Take the good bits of WcfWebApi and the Api development approach of MVC, and created an Api framework to easily expose your REST endpoints and retain the MVC ease of development. This is the WebApi and it supersedes the WcfWebApi (which will not continue to be developed).

So how do you make a REST sandwich now?

Well, best to start with the code.

Firstly, lets define a route for our REST Api.

In the Global.asax, we may have something like this:

public static void RegisterApis(HttpConfiguration config)
{
    config.MessageHandlers.Add(new ApiUsageLogger());

    config.Routes.MapHttpRoute("contacts-route", "contacts", new { controller = "contacts" });
}

Ignore the MessageHandler line for now, we will get back to that in a later post.

You can see we are defining/mapping a Http route. The first parameter is the unique route name, second is the route template to use for processing these route requests. This means that when I get a request like http://MyHost/Contacts, this route will get a match. The third parameter the default properties. In this case, I am simply stating that the controller to use is the ContactsController. This is shown below.

public class ContactsController : ApiController
{
    public List<ContactSummary> GetContacts()
    {
        // do stuff....            
    }
}

You can see that we have a class that inherits from a new ApiController class. We have a simple controller action that returns a list of ContactSummary objects.

This is (while overly simplistic) a fully fledged REST Api that supports content negotiation and as such will respond with json or XML if it is requested by the client.

The WebApi fully supports your typical Http verbs such as GET, PUT, POST, DELETE etc… and in the example above, the fact that I have an ApiController action method prefixed with a ‘Get’ means it will support the GET verb. If I had a method prefixed with ‘Post’ then that action method, by convention, would support the POST verb. You can optionally decorate the methods with [System.Web.Http.HttpGet], [System.Web.Http.HttpPost] attributes to achieve the same effect.

And OData as well?

Want to support OData? Then simply return an IQueryable<T>. Your Web Api will support querying via the OData URL conventions.

So how do I access this as a consumer?

Accessing your shiny new WebApi can now be done via the HttpClient class which really emphasises everything asynchronous, and easily supports the popular Http verbs GET, PUT, POST, DELETE. Let’s take a look.

HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var responseMsg = client.GetAsync("http://SomeHost/Contacts").Result;

The responseMsg variable will contain a list of contacts as shown the in Api method described earlier. You can see we are requesting the data in JSON format. This could also be XML by using:

HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
var responseMsg = client.GetAsync("http://SomeHost/Contacts").Result;

The Http client has methods for using http verbs explicitly. Namely, GetAsync, PutAsync, PostAsync and DeleteAsync.

Summary

Microsoft have recognised the value of the MVC programming model and incorporated the best of WcfWebApi into MVC for a really nice way of exposing your Api. You get a lot of features for free making great Api’s a hell of a lot easier.

You can get further information on the overall MVC4 release from this post by Jon Galloway.

In the next part to this post, I will explore how you can do model binding, dependency injection, and insert custom handlers into the MVC/Api processing pipeline.

ScriptHelper now a Nuget package, and managing your Javascript

Glav — Sat, 21 Jan 2012 03:45:54 GMT

For a while now I have been looking at different ways of managing javascript inclusion in web pages, and also managing the dependencies that each script inclusion requires. Furthermore, working with ASP.NET MVC and partial views, working with the ever increasing number of dependencies as well as ensuring that each partial view has the script it requires, can be a little challenging. At the very least messy and tedious.

Ideally, I’d like every page or partial view to be able to express what scripts it requires and let the framework care about removing duplication, minification etc.

So, after looking at the needs of our application, as well as the needs of others I developed the ScriptHelper component which is now available as a Nuget package. I had released an initial version some time ago, details are here.

This latest version has the following features:

Support for RequiresScriptsDeferred and RenderDeferred methods to allow you to specify script requirements as many times as you like, where ever you like and to have these requirements only rendered to the page when the RenderDeferred method is called. This makes it easy to include the RenderDeferred at the bottom of your master page so all deferred scripts are rendered then. These scripts can be minified and combined at this time as well.

So you can do:

@ScriptDependencyExtension.ScriptHelper.RequiresScriptsDeferred("jQuery")

and maybe somewhere else in the page or in another partial view

@ScriptDependencyExtension.ScriptHelper.RequiresScriptsDeferred("jQuery-validate-unobtrusive")

@ScriptDependencyExtension.ScriptHelper.RequiresScriptsDeferred("SomeOtherScripts")

Then when the:

@ScriptDependencyExtension.ScriptHelper.RenderDeferredScripts()

is called, all the previously deferred scripts are combined, minified and rendered as one single file inclusion

Support for .Less so you can have variables and functions in your CSS. No need to use a .less extension as .Less is automatically invoked if it is enabled for the first time it is required.

Note: Although the project source is titled as an MVC script helper, it can also be used within Webforms without issue as it is a simple static method with no reliance on anything from MVC. I just happened to start coding it with an MVC project in mind.

Additionally, the helper fully supports CSS and .less CSS semantics and syntax.

There are other frameworks that do similar things but I created this one for a few reasons:

The code is pretty small and simple. Very easy to modify and extend to suit your own purposes. It currently uses the Microsoft Ajax minifier. If you dont like it, implement the “IScriptProcessingFilter” and replace the current minifier component with your own.
I liked the ability to express dependencies explicitly. The number of JS libraries to use grows every day and it can get tricky in large apps to easily see what is required and needed.
I didn’t find one that easily implemented the deferred loading scenario, took care of duplicates, and minification, and .less support. Maybe there is now, but I got started on it anyway. Or maybe there is one, but the implementation looked ugly. Either way, I wasn’t satisfied with the current offerings.
I like coding stuff

Download the Nuget package from here.

Download the source from here.

For full documentation on ScriptHelper (minus the new features here), see this post.

A ReadMe.txt file is included in the package with all configuration details.

But .Net 4.5 will include a bundling facility to address some this. Why would I use this?

See Scott Gu’s blog post for more info on this feature.

Well of course you don't have to and if you were happy with the default way bundling works, it is probably worth sticking with that. I mean, you don’t want to include extra libraries if you don’t have to.

However the bundling supports expressing dependencies programmatically in code. it is ok, but I prefer the XML file. Easier to read and define IMO. Sure the code isn’t that hard to read, but it could be anywhere and takes a bit more analysis to see all of the dependencies any one component may need.

There is no support for .less files out of the box. I am sure this wont be far off as adding .less support is pretty trivial. However it is not in there by default.

Bundling will include what you tell it to when you tell it to. This is good, but I love the idea of views or pages being able to express their dependencies without worry of script duplication. The deferred script loading feature of the ScriptHelper was one of my main reasons for developing this. If you don’t use deferred script loading/inclusion, then ScriptHelper will assume you know what you are doing and include the script file immediately as per bundling.

The bundling feature will use .Net 4.5. If you are bound to lower versions of the framework, this is a blocker. ScriptHelper is compiled against .Net 4, however there is nothing that really relies on .Net 4 specifics. If you were targeting .Net 1, it could prove a little tricky as generics are used and a few other features, but migrating/compiling against lower versions should not be very hard.

Managing your Javascript

So how does this help manage your javascript in your projects?

PartialViews

Normally partial views, other pages either rely on a knowledge of what is included via the master page and what is also included on the content page itself in terms of scripts. A partial view may include its own scripts, provided that script is not included elsewhere to prevent duplicate script files being loaded.

Using ScriptHelper and DeferredScripts your partial view can include anything all its dependencies. So the partial view can use the “RequiresScriptsDeferred” method to express its dependencies. At some point in time, typically at the bottom of your master page/layout, you will call the “RenderDeferredScripts” where all the required scripts are rendered. Typically this is also used with script combination and minification so that only 1 include is rendered. All duplicate inclusions are ignored, all files are combined, minified, and .less filter is run over it if required. A single script include is generated and your done. If the master page/layout has already expressed dependencies (such as jQuery for example) and the partial view also expresses this dependencies, it is ignored.

This means each partial view, component or page can express all its dependencies that it requires without fear of duplication.

Script Dependencies

The way ScriptHelper works is by using a friendly name to group one or more script files together in the ScriptDependencies.xml file. The way this is all grouped is up to you. You can group by component (jQuery, validation, etc…) or even by page or anything else. There only needs to be 1 reference to the explicit name of your script, everything else is by friendly name so when a version number of a script changes, potentially changing the name of the script file, you only need to change the name of the file in one place.

In addition, the dependencies for scripts in the XML file are listed explicitly. There is no guesswork as to what script require what in terms of dependencies since it is listed for easy identification.

Finally, if you like having version identifiers on your query strings to assist with browser caching, this can also be expressed in the ScriptDependencies.xml file. This means you can have a script with http://host/script.js?v=123 where the v=123 is the version identifier. The ‘v’ and the ‘123’ are all declared in the XML file which is easily updated manually or via a build script.

So if you find yourself in Javascript inclusion and dependency hell, give this library a shot, it may help.

My Code Kata–A Solution Kata

Glav — Sun, 13 Nov 2011 03:53:56 GMT

There are many developers and coders out there who like to do code Kata’s to keep their coding ability up to scratch and to practice their skills. I think it is a good idea.

While I like the concept, I find them dead boring and of minimal purpose. Yes, they serve to hone your skills but that’s about it. They are often quite abstract, in that they usually focus on a small problem set requiring specific solutions. It is fair enough as that is how they are designed but again, I find them quite boring.

What I personally like to do is go for something a little larger and a little more fun. It takes a little more time and is not as easily executed as a kata though, but it services the same purposes from a practice perspective and allows me to continue to solve some problems that are not directly part of the initial goal. This means I can cover a broader learning range and have a bit more fun. If I am lucky, sometimes they even end up being useful tools.

With that in mind, I thought I’d share my current ‘kata’. It is not really a code kata as it is too big. I prefer to think of it as a ‘solution kata’.

The code is on bitbucket here.

What I wanted to do was create a kind of simplistic virtual world where I can create a player, or a class, stuff it into the world, and see if it survives, and can navigate its way to the exit.

Requirements were pretty simple:

Must be able to define a map to describe the world using simple X,Y co-ordinates. Z co-ordinates as well if you feel like getting clever.
Should have the concept of entrances, exists, solid blocks, and potentially other materials (again if you want to get clever).
A coder should be able to easily write a class which will act as an inhabitant of the world.
An inhabitant will receive stimulus from the world in the form of surrounding environment and be able to make a decision on action which it passes back to the ‘world’ for processing.
At a minimum, an inhabitant will have sight and speed characteristics which determine how far they can ‘see’ in the world, and how fast they can move.
Coders who write a really bad ‘inhabitant’ should not adversely affect the rest of world.
Should allow multiple inhabitants in the world.

So that was the solution I set out to act as a practice solution and a little bit of fun. It had some interesting problems to solve and I figured, if it turned out ok, I could potentially use this as a ‘developer test’ for interviews. Ask a potential coder to write a class for an inhabitant. Show the coder the map they will navigate, but also mention that we will use their code to navigate a map they have not yet seen and a little more complex.

I have been playing with solution for a short time now and have it working in basic concepts. Below is a screen shot using a very basic console visualiser that shows the map, boundaries, blocks, entrance, exit and players/inhabitants.

The yellow asterisks ‘*’ are the players, green ‘O’ the entrance, purple ‘^’ the exit, maroon/browny ‘#’ are solid blocks.

The players can move around at different speeds, knock into each others, and make directional movement decisions based on what they see and who is around them. It has been quite fun to write and it is also quite fun to develop different players to inject into the world.

The code below shows a really simple implementation of an inhabitant that can work out what to do based on stimulus from the world. It is pretty simple and just tries to move in some direction if there is nothing blocking the path.

public class TestPlayer:LivingEntity
{
    public TestPlayer()
    {
        Name = "Beta Boy";
        LifeKey = Guid.NewGuid();
    }
    public override ActionResult DecideActionToPerform(EcoDev.Core.Common.Actions.ActionContext actionContext)
    {
        try
        {
            var action = new MovementAction();

            // move forward if we can
            if (actionContext.Position.ForwardFacingPositions.Length > 0)
            {
                if (CheckAccessibilityOfMapBlock(actionContext.Position.ForwardFacingPositions[0]))
                {
                    action.DirectionToMove = MovementDirection.Forward;
                    return action;
                }
            }
            if (actionContext.Position.LeftFacingPositions.Length > 0)
            {
                if (CheckAccessibilityOfMapBlock(actionContext.Position.LeftFacingPositions[0]))
                {
                    action.DirectionToMove = MovementDirection.Left;
                    return action;
                }
            }
            if (actionContext.Position.RearFacingPositions.Length > 0)
            {
                if (CheckAccessibilityOfMapBlock(actionContext.Position.RearFacingPositions[0]))
                {
                    action.DirectionToMove = MovementDirection.Back;
                    return action;
                }
            }
            if (actionContext.Position.RightFacingPositions.Length > 0)
            {
                if (CheckAccessibilityOfMapBlock(actionContext.Position.RightFacingPositions[0]))
                {
                    action.DirectionToMove = MovementDirection.Right;
                    return action;
                }
            }

            return action;
        }
        catch (Exception ex)
        {
            World.WriteDebugInformation("Player: "+ Name, string.Format("Player Generated exception: {0}",ex.Message));
            throw ex;
        }
    }

    private bool CheckAccessibilityOfMapBlock(MapBlock block)
    {
        if (block == null || block.Accessibility == MapBlockAccessibility.AllowEntry || block.Accessibility == MapBlockAccessibility.AllowExit || block.Accessibility == MapBlockAccessibility.AllowPotentialEntry)
        {
            return true;
        }
        return false;
    }

}

It is simple and it seems to work well. The world implementation itself decides the stimulus context that is passed to he inhabitant to make an action decision. All movement is carried out on separate threads and timed appropriately to be as fair as possible and to cater for additional skills such as speed, and eventually maybe stamina, strength, with actions like fighting. It is pretty fun to make up random maps and see how your inhabitant does.

You can download the code from here.

Along the way I have played with parallel extensions to make the compute intensive stuff spread across all cores, had to heavily factor in visibility of methods and properties so design of classes was paramount, work out movement algorithms that play fairly in the world and properly favour the players with higher abilities, as well as a host of other issues.

So that is my ‘solution kata’.

If I keep going with it, I may develop a web interface for it where people can upload assemblies and watch their player within a web browser visualiser and maybe even a map designer.

What do you do to keep the fires burning?

Updates to the CacheAdapter Package

Glav — Sun, 21 Aug 2011 03:12:52 GMT

Note: This post was originally going to detail the changes from 2.0 to 2.1, however in between the time of release and this post, I released 2.2 so this post will detail all the changes up to 2.2.

In the last update to my CacheAdapter library (you can view it here), I mentioned the support for memcached cache engine. That brought the version to 2.0.

As previously mentioned, you can get this package from here.

I have recently updated the version of the library to ~~2.1~~ 2.2 which now includes the following changes:

The ability provide specific configuration data to a particular cache engine without introducing a bunch of specific configuration tags that are only relevant to a particular cache mechanism.
- Addition of a CacheSpecificData configuration element.
The addition of 2 new simpler API methods to get data from the cache.
- T Get<T>(DateTime absoluteExpiryDate, Func<T> getData)
- T Get<T>(TimeSpan slidingExpiryWindow, Func<T> getData)

Notice that there is no cache key required to be specified. It is auto generated from the Func<T> delegate. (Thanks to Corneliu for this great idea). So you can write really simple code to get data from the cache like so:

var myData = cacheProvider.Get<SomeData>(DateTime.Now.AddSeconds(2), () =>
                                                                          {
                                                                             var somedata = new SomeData();
                                                                             // populate data from somewhere (db, code etc..)
                                                                             return somedata;
                                                                          });

Simplification of the interface
- Replacing the GetDataToCacheDelegate<T>() delegate with a Func<T>
- Removal of the bool addToPerRequestCache = false parameter from the Get<T> methods.

So why the new configuration element? Well, previous versions of this library supported Windows Azure AppFabric however it was only in a non Azure scenario. When Windows AppFabric is used within Azure, a security mode and authentication key is required. This is easily achieved with config, but I wanted a way that did not introduce redundant elements that were not applicable to other cache mechanisms AND that may be useful for other cache mechanisms if they require it. Basically, a generic way of providing configuration data to cache mechanisms, that could be used by any cache mechanism and not pollute the configuration with lots of items specific to only 1 method.

To that end, this release has introduced a “CacheSpecificData” configuration element as shown below:

<setting name="CacheSpecificData" serializeAs="String">
  <value>UseSsl=false;SecurityMode=Message;MessageSecurityAuthorizationInfo=your_secure_key_from_azure_dashboard</value>
</setting>

Currently, this element is only of use to AppFabric cache mechanism, but as support for more cache types grow, this item will be used more and more.

This element is a simple series of key/value pairs that are separated by a semi-colon. In the example above, the following values are defined:

UseSsl = false

SecurityMode = Message

MessageSecurityAuthorizationInfo = your_secure_key_from_azure_dashboard

This equates to the following configuration for Windows AppFabric in Azure (Specifically the <SecurityProperties> element):

<dataCacheClients>
  <dataCacheClient name="default">
    <hosts>
      <host name="somehost.com" cachePort="22233" />
    </hosts>
    <securityProperties mode="Message">
      <messageSecurity authorizationInfo="your_authorisation_key"></messageSecurity>
    </securityProperties>
  </dataCacheClient>
</dataCacheClients>

There are also a few bug fixes and code cleanup in there. Thanks to Corneliu Tusnea (creator of OneSaas) for submitting some really useful changes and bug fixes.

Hope you enjoy the new features.

Go get the Nuget package from here.

Or download the source code from here.

CacheAdapter–V2 Now with memcached support

Glav — Mon, 04 Jul 2011 04:00:34 GMT

Previously I blogged about my CacheAdapter project that is available as a Nuget package and allows you to program against a single interface implementation, but have support for using Memory, ASP.NET Web or Windows AppFabric cache mechanisms via configuration.

I am happy to announce that my CacheAdapter now has support for memcached. Version 2 of the Nuget package is available here. Alternatively, all the source code is available here.

What this now means is you can write one line of code to get or store an item in the cache, and that code can automatically support using Windows AppFabric, memcached, MemoryCache or ASP.NET web cache. No code change required.

I am particularly happy about having memcached support as it means a few things:

Free / Open Source: It is a free, well established open source caching engine that is widely used in many applications.
Easy: It is easy to setup.
Simple and Cheap: It provides an alternative to using Windows AppFabric. AppFabric can be a little tricky to setup sometimes. If you are using Windows Azure, AppFabric is a simple checkbox BUT you need to pay extra for the privilege of using it based on how much you use the cache service. By contrast, memcached can be installed easily on Azure and requires no extra cost whatsoever.
Auto fail over support: In addition, Windows AppFabric has some limitations for a relatively small cache farm. AppFabric utilises a “lead host” to co-ordinate small cache farms of 3 or less cache servers. If the lead host goes down, they all go down. The memcached implementation has no reliance on any single point of failure so if one memcached node fails, requests will automatically be redirected to the nodes that are alive. If the dead node comes back alive again, it is re-introduced to the cache pool after about 1 minute or so. At worst, it results in a few cache misses.

Note: This release contains a few namespace changes that may break older versions if you are using the objects directly. Namely AppFabric object support has been moved from the Glav.CacheAdapter.Distributed namespace to the Glav.CacheAdapter.Distributed.AppFabric namespace. This is to allow differentiation from AppFabric and memcached within the distributed namespace.

In addition, at the request of some users, I have added a simple ‘Add’ method to the ICacheProvider interface for ease of use. The interface now looks like this:

public interface ICacheProvider
{
    T Get<T>(string cacheKey, DateTime absoluteExpiryDate, GetDataToCacheDelegate<T> getData, bool addToPerRequestCache = false) where T : class;
    T Get<T>(string cacheKey, TimeSpan slidingExpiryWindow, GetDataToCacheDelegate<T> getData, bool addToPerRequestCache = false) where T : class;
    void InvalidateCacheItem(string cacheKey);
    void Add(string cacheKey, DateTime absoluteExpiryDate, object dataToAdd);
    void Add(string cacheKey, TimeSpan slidingExpiryWindow, object dataToAdd);
    void AddToPerRequestCache(string cacheKey, object dataToAdd);
}

So that is it. I hope you enjoy using memcached support within the CacheAdapter.

Converting from Webforms view engine to Razor–Some Tips

Glav — Sun, 05 Jun 2011 22:16:55 GMT

Recently, I have had to perform a lot of conversion in an MVC application from using the webforms view engine to the Razor view engine.

I say “had to” but we didn’t really have to, more wanted to. We wanted the cleaner syntax that razor allows. Since we were going to embark on a bunch of new features involving new pages, we wanted to do this conversion earlier rather than later to prevent excessive rework.

This post simply summarises most of the issues and tips that were experienced along the way and how we overcame them, so that others can get a clear picture of what is involved should they embark on such an undertaking, and be prepared for any issues.

Preparation

First off, there are some things you should do prior to starting the conversion to make the process as smooth as possible, and to allow easy resolution of any issues.

Take a copy of your entire source tree and keep it separate to allow for comparison along the way. If you are using HG or GIT, clone your repository to another location. Sure you could look at source control history if you wanted, but its easier to have this code at hand to do side by side comparisons.More importantly though, you can run the backup version to compare HTML output to make sure your conversions are good.
In your solution, turn on MVC compiled views. This will tell the compiler to compile your views and you will catch any errors whenever you compile. Very handy and saves sometime. Unfortunately, it does add some time to the build process. If you have a lot of views, this could be lengthy, so you may want to disable this after the conversion is complete. To enable compiled views:

<MvcBuildViews>false</MvcBuildViews>

and change it to

<MvcBuildViews>true</MvcBuildViews>

Save the file. Then right click on the project again and select ‘Reload Project’
Now when you compile, your views will also be compiled.

Download the aspx2razor conversion tool from Telerik which does a reasonable job of converting your ASPX views into razor views. It probably wont do it all for you,but it removes a good deal of work.

Performing the actual conversion

This is the easy part. Simply run the conversion tool over your project. You can do this by running the tool against your base views directory and have it recursively convert all files it founds in all directories to their razor counterparts. Something like this:

aspx2razor.exe c:\MyProject\MyViews\*.cshtml MyOutputDirectory -r

You will then need to copy the files from the output directory into your solution and ensure they are included in the project. Make sure you delete all the existing ASPX files as having both of these file types present can cause problems.

So that’s it. Job done right?

Testing and Fixing

Now the fun begins. The conversion tool works reasonably well, but it has its problems. Here are some of the issues you will find.

The conversion tool does not support master pages, so don’t expect them to be converted. Roll up your sleeves and do those manually. Even though the tool does not support master pages, it will blindly convert anything you tell it to so it will attempt to convert a master page if one exists in the set of input files. Don’t be tempted to use a partially converted master page if there is one as it will usually not convert properly, putting in an incorrect ‘@inherits’ directive which just causes more confusion. Start from scratch and do the master pages manually.
Sometimes the conversion tool completely misses the server side tags, particularly when embedded in script tags like this:

<script type="text/javascript" language="javascript" src="<%: Url.Content("~/Content/Scripts/jQuery/jshashtable.js") %>"></script>

Embedded server tags can get completely messed up when converting to razor. As an example, the following snippet:

class="item <%: isParentDate ? "" : "child-date " %>
<%: item.IsReconciled ? "reconciled " : "" %>">

gets converted to the following razor equivalent:

class="item @isParentDate ? "" : "child-date "@item.IsReconciled ? "reconciled " : """>

which is obviously incorrect and results in simple text being output to the HTML, not the expected class names. Ideally, this should be refactored into amore explicit code block, something like this:
@{ 
  var divClass = string.Format("item {0}{1}", isParentDate ? string.Empty : "child-date ", item.IsReconciled ? "reconciled " : "");
}
class="@divClass"

“Tight” concatenation of server tags like this:

<input type="hidden" id="itemAmount<%: Model.BankFeed.Id %>" />

get converted to this:

<input type="hidden" id="itemAmount@Model.BankFeed.Id" />

which is not right and just renders the text. Again, something more explicit is required such as:

@{ 
  var controlId = string.Format("itemAmount{0}", Model.Id);
}
    <input type="hidden" id="@controlId" />

One last tip. Use the “original” copy of your site and run it up in a browser, do the same with the converted site. For each view that is converted, it is worth doing a “source view” of original and converted page from the browser to ensure you are getting the same output. This is particularly important for scripts and CSS.

That is about it for most of the issues I came across. You will no doubt experience some variations from the examples above, but the set of issues presented here should at least allow you to be prepared for what the actual conversion effort is going to be.

Hopefully, this article helps you to efficiently convert your projects into the wonderful world of razor

CacheAdapter–Now a Nuget package

Glav — Tue, 31 May 2011 06:12:05 GMT

Note: Link to Wiki is here

A while ago, I blogged about a personal project around caching that allowed you to abstract away the underlying cache mechanism, and use whatever cache you wanted. Through simple configuration, you can choose to use either the memory cache provided by System.Runtime.Caching in .Net 4, the traditional ASP.NET web cache, or you can use the Windows AppFabric caching to taker advantage of distributed caching.

Well, its now a package on Nuget that you can easily install and use immediately. You can grab it from here.

Update: This package uses the Microsoft Public Licence (MS-PL).

Just as a recap, you can use this cache adapter like this:

var data1 = cacheProvider.Get<SomeData>("cache-key", DateTime.Now.AddSeconds(3), () =>
{
    // This is the anonymous function which gets called if the data is not in the cache.
    // This method is executed and whatever is returned, is added to the cache with the
    // passed in expiry time.
    Console.WriteLine("... => Adding data to the cache... 1st call");
    var someData = new SomeData() { SomeText = "cache example1", SomeNumber = 1 };
    return someData;
});

(Note: You will get a small example .cs file included in your project that contains this example usage when you install the package)

You can see in the example, that you can request the data, and provide a lambda that retrieves that data if its not in the cache. The data gets added to the cache once the lambda has been executed to retrieve the data from the source. Also, the data that is returned is strongly typed. It is not a generic object that you then need to cast.

Additionally, you can set the type of cache used by one line of configuration:

<setting name="CacheToUse" serializeAs="String">
    <value>Memory</value>
</setting>

If I wanted to use the ASP.NET Web cache, I could simply set the config item like so:

<setting name="CacheToUse" serializeAs="String">
    <value>Web</value>
</setting>

And if I wanted to use the AppFabric distributed cache (assuming it has been installed on the system), then I can simply set the config like so:

<setting name="CacheToUse" serializeAs="String">
    <value>AppFabric</value>
</setting>

The package references the required AppFabric client assemblies for you. This means you can write a web application today using this package and use the traditional ASP.NET cache. Then we you are ready, install AppFabric, setup a cache cluster, change the configuration to enable AppFabric and you are away. No code changes to your app.

Using AppFabric

To use AppFabric, there are a few extra things to do by way of Infrastructure. Obviously, you need to download and install it. Then you need to setup your cache cluster via the wizard when you install it (or via powershell if you choose). During this process you will be creating some cache nodes and you need to create a named cache for your application. This can be done via powershell using the

New-Cache {cachename}

powershell command. In addition, you will need to ensure that the user that will be accessing the cache has access to it. Again, this can be done from powershell via the:

Grant-CacheAllowedClientAccount {account}

There is some additional config you will need to do as well. In the application config file you will need to change the following entries to match your configured infrastructure:

<setting name="DistributedCacheServers" serializeAs="String">
    <value>localhost:22223</value>
</setting>
<setting name="DistributedCacheName" serializeAs="String">
    <value>MyCache</value>
</setting>

The “DistributedCacheServers” setting can be a comma separated list of servers and ports that the cache client (ie. your app) will communicate with. This can be one or as many comma separated entries as you like and depends on your cache cluster config.

The “DistributedCacheName” is the name of the cache you have setup in the cache cluster for your app. (Detailed above)

Final Notes

Before I published this package, I cleaned it up a bit as I have been using it on a current project and I didn’t like a number of ways I implemented things initially. Instead of multiple assemblies, everything is in the one assembly.

Also, I removed the use of Unity as a mechanism for dependency injection and implemented a very very simple method which uses no particular dependency injection implementation. I did this because everyone wants to do DI (Dependency Injection) using their favourite implementation whether this be Unity, Autofac, Ninject, Castle Windsor etc. etc. So I opted for something that does the job, it is not what a purist would be happy with, but is dead simple to rip out and change which is what I expect most people would do.

Finally, because all cache access is via an interface, it allows for easy testability.

So go grab it. Or you can download the source code form here and change it to your hearts content. Its actually a relatively simple implementation and it would be easy to plug in any underlying cache implementation should you wish.

Diskeeper–a short review

Glav — Sun, 15 May 2011 22:23:40 GMT

I have been lucky enough to be ASP.NET MVP for a number of years now. One of the perks of that is you get access to free licences of software such as Diskeeper. This piece of software keeps your disks running as smooth as they possibly can.

I have been using Diskeeper for around 2 months now and am pretty impressed. It silently does its thing in the background without you having to touch anything. One of the newest and also one of my favourite features is the automatic defrag capability.You don’t have to launch a defrag application and have it run on your system or schedule it to run periodically, as Diskeeper performs this operation all the time, in the background for all disk operations. Basically, you just use your system like normal, and it just magically gets defragmented, all the time. Nice.

I have 2 disks in my system. A SSD (Solid State Drive) as my primary ( C: ) drive, and a 5200rpm 320Gb hard disk as my secondary data drive ( D: ). After installing Diskeeper, I have noticed a significant difference in the performance of the secondary drive in particular. I imagine because it is the slowest disk and because it stands to benefit the most. The SSD is fast anyway, so it is much harder to notice any difference. I have noticed a small difference, but it’s only slight compared to the secondary ( spinning rust ) drive.

I should mention that in order for Diskeeper to work with SSD hardware, you need a special edition of Diskeeper with an add-on called Hyper-fast that takes into account the SSD unit and has special algorithms that are designed to optimise SSD operation.

For those interested in the actual metrics, Diskeeper provides an easy to use management console that displays your current system improvements. The initial display is shown below.

The management console offers a number of ways to look at what has been done to your disks and how it has been performing. I find the history view particularly interesting as it proves a look at how the disk is being used and optimised over time.

So would I recommend it? Yes, and this is not simply because I have been given a free licence and feel obligated to do so. It is particularly useful if you have standard hard disks regardless of how fast they spin. You will notice a much smoother and responsive system as a result. For SSD owners, it is still worthwhile but the benefits are less because of the inherent speed in SSD units. For a system with a mix of SSD and hard disks, it works beautifully.