This blog has moved!

My blog has moved to http://davidfowl.com/
Posted by davidfowl with no comments

SignalR 1.1 beta

We released SignalR 1.1 yesterday with a bunch of performance improvements and new features (detailed in the release notes). Among these new features, were improvements to the .NET client including the ability to set custom headers, client certificates and a custom JSON serializer. We also fixed a long standing issue people were experiencing in IE7 and IE8 with the loading bar showing up forever in the browser. However, most of this release focused on rewriting SignalR’s scaling out story from the ground up.

Scaling out

The previous releases of SignalR’s scale out implementation did not handle a lot of scenarios when it came to scaling it across web servers. We took a step back and redesigned the core scale out architecture and backplanes for SQL Server, Azure Service Bus and Redis.

There are now 3 new packages on NuGet that you can choose from when scaling out SignalR:

  1. SQL Server
  2. Azure Service Bus
  3. Redis

These backplanes share a common base class (ScaleoutMessageBus) that contains the core logic allowing SignalR to work across a webfarm.

Using Scale out

Registering a backplane hasn’t changed much from the previous versions but there are some new features to be aware of.

ALL backplanes

Scale out backplanes have 2 modes controlled by a single flag RetryOnError. Users can create a configuration object for their backplane and specify which mode they want to use. This flag determines if SignalR should start buffering messages when the scale out backplane encounters errors; by default this flag is false. Here is an example:

 

 

In the above scenario, if the Redis service went offline, with RetryOnError = false the call to send a message to all clients would throw synchronously. If RetryOnError was true, the OnError callback would be raised the call would be queued until Redis came back online. This queue does have a limit and when it is full it will also fail.

 

SQL SERVER

  • The SQL backplane requires that you create the database before hand.
  • The SQL scale out backplane works with or without service broker turned on.
  • Performance is usually better with service broker on since this enables query notifications to work.

Once the database is set up, you can enable SQL like this:

It’s all you need to get running with SQL server.

 

SERVICE BUS

The new service bus scale out backplane only works with the Windows Azure service. The wiki has been updated with new steps that show how to setup service bus on Windows Azure.

Windows Azure Websites is also supported.

TRACING

If you are encountering problems with scale out and want to troubleshoot what’s going on, you can turn on tracing with this xml snippet in your config file:

 

 

The purpose of this beta release is to get feedback, so do give it a try and let us know what you think. As always, file issues on github, and questions on stackoverflow, jabbr or the SignalR forum.

Posted by davidfowl with 2 comment(s)

Microsoft ASP.NET SignalR

Yes, you read that right, Microsoft.AspNet.SignalR. It’s been a while since I last blogged as we’re super busy trying to make SignalR rock solid and 1.0 quality for you all to use. I’ve also given my first talk ever at Monkeyspace and then BUILD - both on SignalR. SignalR 1.0 alpha has been live for just over 2 weeks on NuGet, and a few seconds ago I just published alpha2. A lot has changed since 0.5.3 and there have been several people poking around trying to find out what’s changed and what’s new in the alpha. There’s also been lots of confusion about what to use, so I’m going to clear things up by enumerating the list of major changes.

New Packages

There are new packages that you can use today in your applications.

  • Microsoft.AspNet.SignalR – meta package (use this)
  • Microsoft.AspNet.SignalR.Client – .NET 4 and WinRT client
  • Microsoft.AspNet.SignalR.JS – The Javascript client.
  • Microsoft.AspNet.SignalR.Core – Core server package with no host implementation
  • Microsoft.AspNet.SignalR.Hosting.AspNet – The ASP.NET host
  • Microsoft.AspNet.SignalR.Hosting.Utils – utilities for signalr (signalr.exe)
  • Microsoft.AspNet.SignalR.Redis – Redis message bus implementation
  • Microsoft.AspNet.SignalR.ServiceBus – Service bus message bus implementation

If you’re using the NuGet dialog, pick the IncludePrerelease in the dropdown to see them. If you’re using the package manager console, use the –pre flag to install it.

Install-Package Microsoft.AspNet.SignalR –pre

Visual Studio Integration

When you install the ASP.NET 2012 Fall Update, you’ll see a few new things in your Add New Item menu.

VsIntegration

These will bring in the alpha1 packages but you can update them using NuGet to get the latest and greatest.

Hub API changes

First thing to notice is the client and server members on the hub object. This came about because there was a subtle issue with naming methods on the client and server. If they were called the same thing it just wouldn’t work so we decided to separate the namespaces. As you already guessed, you can declare methods that the server calls on the client object and call methods that exist on the server with the server object.

On the server side we’ve added quite a few new APIs for hubs after lots of feedback and discussion.

This is a dump of all the new surface we’ve added for hubs. The comments are pretty self explanatory but as part of this work we resolved this long standing feature request https://github.com/SignalR/SignalR/issues/105 (it’s almost a year old!).

Connected, Disconnected and Reconnected

Before we had these awful interfaces you had to implement to detect this, now it’s just an override on the Hub itself.

No more magic?

There’s obviously still some magic, just a bit less than before. When using hubs, people always wondered how this magic ~/signalr/hubs url was getting registered. This is now an explicit call the developer has to make to enable this route. When you install the new package, it’ll add a RegisterHubs.cs file to the project in the App_Start folder.

There's even a little comment telling you what MapHubs does. If you want to, you can remove this file and just add the call to your global asax with the rest of your route registration.

Hub Pipeline

This is the most powerful feature we’ve added in the new release. Think of the hub pipeline like mvc filters but more powerful. You get to inject any piece of code anywhere in the hub pipeline for incoming and outgoing calls. Here’s an example of a hub pipeline module that logs all outgoing and incoming calls.

The module has a bunch of different methods you can override, the above sample only shows 2 of them. Each method takes the appropriate context so you can get whatever information you want about the incoming or outgoing invocation. The next feature on the list builds on the pipeline implementation.

Authorization

Many of you wanted an easy way to mark a hub as requiring authorization and now you can do that by just specifying an attribute similar to MVC.

Now you can decorate your hubs or hub methods or both to get different granularities of security. The first example shows requires authorization for all connections to the hub and calls to any incoming methods. In the second example, any connection to the hub as well as calls to Send require authorization, but calls to Send2 do not.

Client side keep alive

This feature allows clients to react to slow/buggy networks that drop connectivity often. The JavaScript client checks for a keep alive ping from the server on a configurable interval and raises the “connection slow” event before it drops the connection and starts reconnecting.

Rejoining Groups

There’s tons of confusion about how groups work in SignalR and how it works in scaled out environments (http://stackoverflow.com/questions/11143220/signalr-groups-filtering-handled-on-client-or-server). SignalR stores enough state on the server to make groups work, but all groups that a client is a member of is round-tripped with that client. This of course means that any client can claim they are in a group and spy on your messages. Groups are not a security feature, it’s about message filtering. By default, SignalR will ignore groups that are sent from client to server (to be secure by default). Clients will not be able to receive messages for groups they were in after a reconnect for most transports, but for longpolling it will be immediate. Developers can re-enable the default behavior by enabling a pipeline module that trusts the client’s groups.

If you’re more advanced and want to apply a custom filter to pre-process groups for particular clients, you can write a custom pipeline module to do so.

New Scale Out Providers

We’ve completely re-worked the scale out providers for service bus and redis and added sql server to the list (not shipping with this release but you can use the source). As part of this work, we’ve abstracted a ScaleOutMessageBus base class that each of these providers derive from in order to cleanly plug into SignalR. Check out the class for more details.

More Memory Efficient and huge throughput gains

The MessageBus was complete re-written and optimized for small messages (we normally test with 32 byte sized message payloads) and high throughput. More memory efficient means that we can get more connections on a single node. By the time we RTM, we’ll have documentation and guidance on how to do capacity planning on your webservers and on azure.

Performance Counters

We have a utility called signalr.exe (in the Microsoft.AspNet.SignalR.Hosting.Utils package) that you can use to do a few things.

  • Install/Uninstall SignalR performance counters
  • Generate a static js file for the magic ~/signalr/hubs url.

You can install the performance counters if you want to see more detailed information about your SignalR applications. Here’s a view of what that looks like:

Counters

 

NOTE: To use these on full IIS you need to add the app pool user for your application to the Performance Monitor Users group.

Message payload size

We’ve made some changes (not in alpha2 but later) to reduce the size of our payload dramatically and as a result changed the protocol for 1.0 as well. I’ll talk about this more in a later post.

Documentation

We’re in the process of updating it as well as preparing content for http://www.asp.net/signalr. Stay tuned!

THANK YOU ALL

Thank you all for being a part of this amazing journey from side project to official project. Thanks to your enthusiasm it has has grown rapidly into a framework that people just love to use! Keep giving us feedback! Ping me on jabbr or twitter if you have any questions!

Posted by davidfowl with 24 comment(s)

SignalR 0.5.1 Released

Here comes another exciting release of SignalR. This release has some new features that you can take advantage of as well as a few bug fixes. Let’s go over the big changes for this release.

Windows 8 Client support

With the Windows 8 Release Preview, we’ve finally added a new client to support “.NET for Metro Style Apps” and also made some fixes so that SignalR’s JavaScript client works with WinJS.

WebSocket Support

Previously, websocket support required a separate package, now it’s built into SignalR. The ASP.NET host is the only one that supports this and that’s only when running an application targeting .NET 4.5 on IIS8.

Improved Cross Domain Support

We introduced cross domain support for SignalR in 0.5. In this release we’ve improved that support in a few ways:

  1. All SignalR end points are CORS enabled by default. This means you can point the JavaScript client to the endpoint and have it just work without having to manually add the cors header to web.config.
  2. SignalR will choose automatically negotiate the transport so there’s no need to specify it or the xdomain flag (which is no longer supported).
$.connection.hub.url = 'http://localhost:8081/signalr;

$.connection.hub.start();

The above code is all you need to do to connect to a remote server using SignalR.

Connection State Changes

A SignalR connection has always been a black box to the user and this made it hard to detect changes in the underlying connection state. We’ve exposed a new stateChanged event in the JavaScript and .NET clients. This allows you to listen for state changes and react to them in different ways. Here’s an example that shows the user a message if the connection went into a reconnecting state and didn’t recover after 10 seconds:

var chat = $.connection.chat;
var timeout = null;
var
interval = 10000;

chat.addMessage =
function
(msg) {
    $(
'#messages').append('<li>' + msg + '</li>'
);
};

$.connection.hub.stateChanged(
function
(change) {
   
if
(change.newState === $.signalR.connectionState.reconnecting) {
        timeout = setTimeout(
function
() {
            $(
'#state').css('backgroundColor', 'red'
)
                        .html(
'The server is unreachable...'
);
        }, interval);
    }
   
else if
(timeout && change.newState === $.signalR.connectionState.connected) {
        $(
'#state').css('backgroundColor', 'green'
)
                        .html(
'The server is online'
);

        clearTimeout(timeout);
        timeout =
null;
    }
});

$.connection.hub.start();

When the connection state changes to reconnecting, we set a 10 second timer, if the state doesn’t change to “connected” in those 10 seconds then we set the message to say the server is unreachable. When the connection comes back online (i.e the state the connected), we set the message, “The server is online”.

This feature should give app developers a chance to deliver nicer user experiences when for clients on crappy connections or when servers go down unexpectedly. We’ll continue to improve this in the future.

Clean Disconnect

Normally when a connection is broken between the browser and server, the underlying http request takes some time to go away (you can use fiddler to see this). In 0.5.1 we’ve added the ability for the client to send a packet to the server whenever a connection is stopped or the browser is closed. This means disconnect will fire more immediately on browser close or on page refreshes.

Hubify.exe

It’s not a nuget package but will be in the http://chocolatey.org/ gallery soon. For now, you can run get hubify.exe by building https://github.com/SignalR/SignalR/tree/master/SignalR.ProxyGenerator. You can run this against your project’s bin directory or a remote url to generate or download the hubs.js file that would normally be dynamically generated when referencing the magic script url:

<script src="signalR/hubs" type="text/javascript"></script>

You can add this to a post build event in your project to generate the file a build time so that you can take advantage of http caching when you deploy your application (See an example here).

Keep Alive

Keep alive is now on by default and is set to 30 seconds.

Those are the highlights, view the release notes for more information.

Get it on NuGet today!

Posted by davidfowl with 11 comment(s)

API Improvements made in SignalR 0.5

We made lots of breaking changes in 0.5 in order to gain more API consistency across the board. Some of the changes may not affect you if you weren’t doing anything with lower level APIs. These types of breaking changes will obviously slow down as we get closer to a 1.0 release.

API comments

This release, we spent some time writing API comments that should be helpful during development.

Breaking changes

A bunch of APIs were removed from PersistentConnection including the following:

  • All overloads of Send
  • SendToGroup
  • AddToGroup
  • RemoveFromGroup

Sending Data

All data is now sent through an IConnection (Connection property). Now, whether you’re within a PersistentConnection or outside of one, the logic to send data over that connection will look exactly the same.

SignalR 0.4

public class MyConnection : PersistentConnection
{
   
protected override Task OnConnectedAsync(IRequest request, IEnumerable<string> groups, string
connectionId)
    {
       
return Connection.Broadcast("A new connection was made!"
);
    }

   
protected override Task OnReceivedAsync(string connectionId, string
data)
    {
       
return Send(data);
    }
}

SignalR 0.5

public class MyConnection : PersistentConnection
{
   
protected override Task OnConnectedAsync(IRequest request, string
connectionId)
    {
       
return Connection.Broadcast("A new connection was made!"
);
    }

   
protected override Task OnReceivedAsync(string connectionId, string
data)
    {
       
return Connection.Send(connectionId, data);
    }
}

Groups

Group management is now centralized and consistent across Hubs and PersistentConnections. Each of those types have a property Groups of type IGroupManager that are scoped to a specific PersistentConnection or Hub. Here’s an example of adding a group in 0.4 and 0.5:

SignalR 0.4

public class MyConnection : PersistentConnection
{
   
protected override Task OnConnectedAsync(IRequest request, IEnumerable<string> groups, string
connectionId)
    {
       
return AddToGroup(connectionId, "foo"
);
    }

   
protected override Task OnReceivedAsync(string connectionId, string
data)
    {
       
return SendToGroup("foo", data);
    }
}
public class MyHub : Hub
{
   
public Task Join(string
group)
    {
       
return
AddToGroup(group);
    }

   
public Task Send(string group, string
message)
    {
       
return Clients[group].addMessage(message);
    }
}

SignalR 0.5

public class MyConnection : PersistentConnection
{
   
protected override Task OnConnectedAsync(IRequest request, string
connectionId)
    {
       
return Groups.Add(connectionId, "foo"
);
    }

   
protected override Task OnReceivedAsync(string connectionId, string
data)
    {
       
return Groups.Send("foo", data);
    }
}
public class MyHub : Hub
{
   
public Task Join(string
group)
    {
       
return
Groups.Add(Context.ConnectionId, group);
    }

   
public Task Send(string group, string
message)
    {
       
return Clients[group].addMessage(message);
    }
}

Dependency Injection

We’ve removed the AspNetHost type that from SignaR 0.4 and added a type that is purely optional to use, GlobalHost. The idea behind this type is to allow easy access to items in the container that will be used most commonly in any SignalR application, as well as a host agnostic default dependency resolver. This type exists in a new assembly SignalR.Hosting.Common.dll.

Replacing the resolver

SignalR 0.4

public class Global : System.Web.HttpApplication
{
   
protected void Application_Start(object sender, EventArgs
e)
    {
       
AspNetHost.SetResolver(new CustomResolver());
    }
}

SignalR 0.5

public class Global : System.Web.HttpApplication
{
   
protected void Application_Start(object sender, EventArgs
e)
    {
       
GlobalHost.DependencyResolver = new CustomResolver
();

       
RouteTable.Routes.MapHubs();
    }
}

Here, we’re changing the global resolver and remapping the default hub route to pick it up. The hub route is automatically added at a very early stage in the application (PreApplicationStart) so we need to remap it to use the new default resolver or we can specify a new one for the hub route directly:

public class Global : System.Web.HttpApplication
{
   
protected void Application_Start(object sender, EventArgs
e)
    {
       
RouteTable.Routes.MapHubs(new CustomResolver());
    }
}

The above logic doesn’t replace the global resolver but instead creates a self contained SignalR world using the specified resolver. If you go with this more “pure” approach, you’re opting out of using the GlobalHost.

Changing the Hub route

In 0.4, the magical signalr/hubs url was implemented as a dynamic http module. Now it’s just a regular route in the route table that can be replaced. Here’s an example of changing it to signalr2:

public class Global : System.Web.HttpApplication
{
   
protected void Application_Start(object sender, EventArgs
e)
    {
       
RouteTable.Routes.MapHubs("~/signalr2");
    }
}

Obviously this change needs to be accompanied by a change to your JavaScript include:

<!DOCTYPE html>
<
html
>
<
head>
    <title></title>
    <script src="Scripts/jquery-1.6.4.js" type="text/javascript"></script>
    <script src="Scripts/jquery.signalR-0.5rc.js" type="text/javascript"></script>
    <script src="signalr2/hubs" type="text/javascript"></script
>
</
head
>
<
body>

</body
>
</
html
>

Getting a reference to the connection/hub outside of a connection or hub

This is an extremely popular pattern for integrating with other frameworks. Here’s an example of broadcasting to all clients from an mvc controller:

SignalR 0.4

public class HomeController : Controller
{
   
public void
PerformLongRunningOperation()
    {
       
var connection = AspNetHost.DependencyResolver.Resolve<IConnectionManager>().GetConnection<MyConnection
>();

        connection.Broadcast(
"Called from an mvc controller"
);
    }

   
public void
PerformLongRunningHubOperation()
    {
       
var clients = AspNetHost.DependencyResolver.Resolve<IConnectionManager>().GetClients<MyHub
>();

        clients.notify(
"Hello world");
    }
}

SignalR 0.5

public class HomeController : Controller
{
   
public void
PerformLongRunningOperation()
    {
       
IPersistentConnectionContext context = GlobalHost.ConnectionManager.GetConnectionContext<MyConnection
>();

        context.Connection.Broadcast(
"Called from an mvc controller"
);
    }

   
public void
PerformLongRunningHubOperation()
    {
       
IHubContext context = GlobalHost.ConnectionManager.GetHubContext<MyHub
>();

        context.Clients.notify(
"Hello world");
    }
}

With this, you can inject these new context objects into your mvc controller or whatever framework you want to combine with SignalR. For example:

public class HomeController : Controller
{
   
private readonly IHubContext
_hubContext;

   
public HomeController(IHubContext
hubContext)
    {
        _hubContext = hubContext;
    }
       
   
public void
PerformLongRunningHubOperation()
    {
        _hubContext.Clients.notify(
"Hello world");
    }
}

Configuring SignalR

On each host (Self host and GlobalHost) there’s a Configuration property. This gives you access to the server configuration for things like setting timeouts, the keep alive interval and the heart beat interval:

public class Global : System.Web.HttpApplication
{
   
protected void Application_Start(object sender, EventArgs
e)
    {
       
GlobalHost.Configuration.KeepAlive = TimeSpan.FromSeconds(20);
    }
}

I’ll be blogging about more of these changes as we get closer to the release of 0.5.

Posted by davidfowl with 9 comment(s)

SignalR 0.5

This is the rebirth of my blog. I’ve been busy over the last few months with projects like JabbR, SignalR and other stuff. With the upcoming release of SignalR 0.5, I thought it’d be good to write a blog post that touches on some of the things we did.

Webfarm Support

There’s a lot of confusion as to what webfarm support means in SignalR. This is an FYI, SignalR 0.5 will not automatically make your application work on azure with zero effort. SignalR needs a way to communicate between instances of your webserver, in 0.5 we implemented 2 of those providers on NuGet (message buses):

  1. SignalR.WindowsAzureServiceBus - http://vasters.com/clemensv/2012/02/13/SignalR+Powered+By+Service+Bus.aspx (blog post slightly outdated but information is relevant).
  2. SignalR.Redis – Redis message bus implementation for SignalR - https://github.com/SignalR/SignalR.Redis

When deciding to make SignalR work in a webfarm, you’ll have these 2 options. We’re going to continue improving them as well as implement solutions on top of popular message buses like NServiceBus and lower fidelity solutions built on top of SQL QNS and possibly SQL Server Service Broker.

This wasn’t the only hurdle to making SignalR work in webfarms, there were some nasty bugs that caused some bad behavior that we fixed:

  1. Disconnect notifications across webfarms
  2. Load balancer idle timeout in azure.

Fixing disconnect in webfarms was the most important issue to fix. Before SignalR 0.5rc, disconnect in a webfarm (using service bus or redis) could fire for the same client multiple times. This is more obvious when using longPolling and load balancing happens between web server instances. If you had control of your LB, you could affinitize clients to a server so you’d never see this behavior, but I’m glad to say we got this fixed.

For load balancer idle timeouts (especially on azure), we support keep-alive. You can now tell SignalR to send an empty packet on a specific time interval. This will fend off nosy network devices that like to kill idle connections waiting for data. This is especially important for things like server sent events and forever frame transports.

Mono Support

In the 0.4 release, SignalR only “worked” when self hosted on mono so I spent time making SignalR work across the board (ASP.NET and all). When I say work, I mean both the client and the server. You can now literally clone SignalR onto your OSX (I didn’t try linux) box and run make and get a successful build (see the README for more details). This wasn’t a simple task as I had to make changes to the NuGet targets file so that package restore worked on mono.

If you’re using Mono develop, open SignalR.Mono.sln. You can run the samples as you would normally. Here’s a glimpse of the chat demo running:

551939760

 

Not much time was spent optimizing and tuning specifically for mono but in the future we’ll have time to do that.

JsonP Longpolling support (XDomain)

This feature wasn’t planned for 0.5 but thanks to an awesome contribution from codeputty, this became a reality.

Dynamic Hubs Implementation

Contributor pszmyd was trying to build better integration between SignalR and Orchard and ended up adding infrastructure to SignalR to decouple Hubs from types (System.Type). This is a great example of why I love OSS development (scratch your own itch).

Continuous Integration and NuGet Server

We finally got a CI server setup on codebetter’s teamcity instance. You can view SignalR here http://teamcity.codebetter.com/project.html?projectId=project188. The CI also builds packages and pushes them to our myget feed. If you want to live on the bleeding edge but don’t want to build from source, you can configure nuget to point to http://www.myget.org/F/signalr/ (make sure you use the –Pre flag or select the Include Prerelease in the dialog).

 

0.5 Release

Help us test the RC now on NuGet! (http://signalr.net/releases/v0584.html). We’re reserving this week for any bugs that we consider ship stoppers.

What’s Next

We’re going to continue improving SignalR. Future releases will focus on some of the following:

  • Performance Improvements
  • In the box WebSockets support for Windows8
  • Windows 8 .NET Client
  • .NET Micro Framework Client
  • Web API Host
  • Build time hubs.js generation (instead of runtime).
  • Better network availability handling
  • Better Mobile support (optimized for battery life)
  • Web Forms control

… And much more! Check the issue backlog for details on what else we've got planned for upcoming releases.

Come Talk with us

We’re always around in the signalr room on jabbr. Come say hi!

Posted by davidfowl with 7 comment(s)

Razor Debugger for ASP.NET WebPages

I have neglected blogging recently because I’ve been super busy working on Razor tooling, NuGet and Beta 3 Release of the Web Stack. Now that I have some free time, I can blog about a really cool project I’ve been working on.

The Razor Debugger is a simple web based debugger for ASP.NET WebPages (it might work in MVC but hasn’t really be tested). It started off as an experiment to see if it was possible to have a simple debugging experience for developers using WebMatrix.

It’s a NuGet Package

Of course the Razor Debugger is distributed as a NuGet package and is available today on the WebMatrix feed. You can learn more about ASP.NET WebPages package management here.

Debugging Tutorial

Let’s step through a simple example on how to use the debugger with your site.

Start by opening WebMatrix and choose Site From Template. Create a new Empty Site and create a new cshtml page.

image

Next, we’re going to install the RazorDebugger through the admin UI. Launch the website and navigate to the _Admin url in your browser:

image

Type your secret password and continue to the Package Manager. List all packages from the online feed and install the RazorDebugger package:

image

After installing the debugger, navigate to the debugger page by going to ~/debugger.

image

The debugger is a section of the website that shows all of the pages on the left, and some UI on the right that we’ll describe in further details below.

To start debugging your site, put some code in the page and click on the page in the debug view.

image

Now that we have some code in our page we can set a break point by clicking on the gray pane beside the line numbers in source view. When a break point is set, a red circle will be placed on the line where the break point was set.

Start debugging! Click on Launch Page at the top of the page to run the page. Switch back to the debugger and see the break point being hit.

image

Now that we’re debugging, we can use the Step Into, Step Over, Step Out and Run buttons or the associated shortcut keys. Also, as you step, the locals, watches and stack trace UI at the bottom will change to show locals and watches respectively.

image

You can also change the values of locals by clicking on the value of the local and typing a new one.

image

Stepping into other pages

It’s possible to step into other pages if a call to RenderPage is made within the page you are debugging. I’ve updated the sample to contain 2 pages, Page.cshtml passes an anonymous type with the Name “David” to Page2.cshtml.

image

Now we can hit Step Into or i and step into Page2.cshtml.

image

Exceptions

The debugger also supports unhandled exceptions. The page below shows an example of a NullReferenceException. When an exception occurs the line is highlighted with a dotted border (it’s kind of hard to see) and the exception object shows up in the locals window.

image

Start Debugging

This debugger is a prototype and isn’t meant to replace the debugger in visual studio by any means. It is however a pretty cool experiment and you can try it out today in your ASP.NET WebPages!

Posted by davidfowl with 5 comment(s)

Dynamic LINQ Part 2 (Evolution)

A while ago, Scott Gu blogged about a Dynamic LINQ library that was part of the C# samples in VS2008. This library let developers put together late bound LINQ queries using a special syntax. This really comes in handy when you want to build a LINQ query based on dynamic input. Consider the following example:

public ActionResult List(string sort) {
    var northwind = new NorthwindEntities();
    IQueryable<Product> products = northwind.Products;
    switch (sort) {
        case "ProductName":
            products = products.OrderBy(p => p.ProductName);
            break;
        case "UnitsInStock":
            products = products.OrderBy(p => p.UnitsInStock);
            break;
        default:
            break;
    }

    return View(products.ToList());
}

In the above example, we’re trying to sort a list of products based on the sort parameter before we render the view. Writing the above code is painful, especially if you have a large number of sortable fields. Of course this isn’t the only way to write this code, in fact, if you’re a LINQ expert you can build an expression tree at runtime that represents the sort expression, but that solution is far from trivial to implement.


Last time we showed how we can (ab)use C# 4 dynamic feature to enable LINQ queries against dynamic types. Today I’ll show you how we can expand on that concept, and turn our previous solution into a general purpose library that allows us to execute late bound queries against existing LINQ providers.

The library itself is pretty small, it consists of 2 classes:

  • DynamicExpressionBuilder – The dynamic object that we used last time to build the actual expression tree
  • DynamicQueryable – a static class with a bunch of extension methods that are parallel to the actual Queryable static class

DynamicExpressionBuilder uses the same trick as last time (dynamic dispatch to build expressions), but this time we’re building expressions that would be recognized by most LINQ providers.

public override bool TryGetMember(GetMemberBinder binder, out object result) {
    result = new DynamicExpressionBuilder(Expression.Property(Expression, binder.Name));
    return true;
}

public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
    result = MakeBinaryBuilder(Expression, binder.Operation, arg);
    return true;
}

public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {           
    MethodInfo methodInfo = null;           
    if (args.Any(a => a == null)) {
        methodInfo = Expression.Type.GetMethod(binder.Name);
    }
    else {
        var types = args.Select(arg => arg.GetType()).ToArray();
        methodInfo = Expression.Type.GetMethod(binder.Name, types);
    }           

    if (methodInfo != null) {
        var expression = Expression.Call(Expression, methodInfo, args.Select(Expression.Constant));
        result = new DynamicExpressionBuilder(expression);
        return true;
    }
    return base.TryInvokeMember(binder, args, out result);
}
The actual implementation overrides more methods, but for brevity we’re only going to look at 3 of them. Each method captures the intent of the code 
and turns it into the appropriate
expression tree.
public static class DynamicQueryable {            
    public static IQueryable Where(this IQueryable source, Func<dynamic, dynamic> predicate) {
        return Where(source, GetExpression(source.ElementType, predicate));
    }

    private static LambdaExpression GetExpression(Type elementType, Func<dynamic, dynamic> expressionBuilder) {
        ParameterExpression parameterExpression = Expression.Parameter(elementType, expressionBuilder.Method.GetParameters()[0].Name);
        DynamicExpressionBuilder dynamicExpression = expressionBuilder(new DynamicExpressionBuilder(parameterExpression));
        Expression body = dynamicExpression.Expression;
        return Expression.Lambda(body, parameterExpression);
    }

    private static IQueryable Where(IQueryable source, LambdaExpression expression) {
        return source.Provider.CreateQuery(Expression.Call(typeof(Queryable),
                                                            "Where",
                                                            new[] { source.ElementType },
                                                            source.Expression,
                                                            Expression.Quote(expression)));
    }
    …etc
}
Normally when you’re writing LINQ queries, the implementations of Where, Select, SelectMany and other extension methods come from either the Enumerable or Queryable 
class.The DynamicQueryable class uses this pattern to provide overloads of the same extension methods on the non generic IQueryable.
This lets us do a lot of cool things with existing LINQ providers. Let’s look at some examples:
using System;
using System.Linq;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
Above we have a regular LINQ to Entities query against Northwind. Notice the usings are System, and System.Linq, those make sure that you’re using the Queryable 
implementations of Where and OrderBy.
We can write this same code in with Dynamic LINQ:
using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
We replaced the System.Linq using with a DynamicLINQ using, which makes use of our extensions methods. If we wanted to sort by a dynamic variable you could do it 
using the indexer syntax supported by DynamicQueryBuilder:
using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p["ProductName"]);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
That’s all pretty cool, but there are more scenarios that can’t be done today using regular LINQ which Dynamic LINQ makes easy. 
One of the biggest annoyances for me using LINQ today, is the fact that custom functions can’t be used inside of query expressions. This makes total sense once 
you understand how LINQ expressions work, but if it were possible, you could make big queries a lot be more maintainable.
Take the following query using LINQ to Entities:
using System;
using System.Linq;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => IsCheap(p));

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }

    public static bool IsCheap(Product product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}
Anyone who has done any programming with LINQ to Entities knows that this won’t work. When we try to run this code, we get an error saying that this expression 
cannot be converted to SQL, or more precisely:

Unhandled Exception: System.NotSupportedException: LINQ to Entities does not recognize the method 'Boolean IsCheap(Sample.Product)' method, and this method cannot be translated into a store expression

The reason this doesn’t work is because the compiler puts the IsCheap method call in the expression tree and EF tries to convert it into SQL:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => IsCheap(p))

What we really wanted is for the compiler to replace IsCheap with the expression in the body of IsCheap, i.e. replace IsCheap with !product.Discontinued && product.UnitsInStock < 10.

This same expression would work today with DynamicLINQ:

using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => IsCheap(p));

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }

    public static dynamic IsCheap(dynamic product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}

We tweaked the method slightly, changing the argument and return type to be dynamic. When we run this code it works and we see a list of cheap products.

The difference here is that we’re actually executing code to build the expression tree, so instead of putting the method call into the resultant tree we actually invoke the method and the expression tree ends up looking like this:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => (Not(p.Discontinued) And (Convert(p.UnitsInStock) < 10)))

There are also some scenarios that don’t work in the current implementation. For example, casting and some projections don’t work well with existing LINQ providers. I’ve put the code on a bitbucket repository.

http://bitbucket.org/dfowler/dynamiclinq

Enjoy!

Posted by davidfowl with 10 comment(s)
Filed under: , ,

Dynamic LINQ (A little more dynamic)

It seems I’ve become (in)famous in the last two days. With all the buzz around Microsoft.Data, I figure I’d post on something else that is somewhat related.

There were a bunch of comments on my previous posts that said we should do something to this effect:

dynamic db = Database.Open("name");
var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;
Which looks like LINQ but it would be over a dynamic object. Of course you don’t get the strong typing benefits of LINQ since there are no classes, but at least the 
syntax is nice and the user is protected from SQL injection.

Disclaimer

This blog post is a technical discussion about the limitations that exist today around LINQ and dynamic and interesting ways to get around it. Nothing more. Continue reading.

Unfortunately when we try to compile this the compiler screams:

Query expressions over source type 'dynamic' or with a join sequence of type 'dynamic' are not allowed

Maybe it’s better if we just use a real ORM now, or the other Dynamic LINQ library, but now I’m curious and want to figure out if this is even possible. Let’s try it another way:

dynamic db = Database.Open("name");
var q = db.Products.Where(p => p.UnitsInStock < 20);

The compiler complains again, but this time with a different error:

Cannot use a lambda expression as an argument to a dynamically dispatched operation without first casting it to a delegate or expression tree type

So we’re being blocked in two different ways. Can we work around this somehow?

Since we can’t query over a dynamic object we’ll give the Database and the Products property static types:

class Database {
    private string _databaseName;
    public Database(string databaseName) {
        _databaseName = databaseName;
    }

    public static Database Open(string name) {
        return new Database(name);
    }

    public Query Products {
        get {
            return new Query();
        }
    }
}

class Query {

}

Now we can do this:

var db = Database.Open("name");
var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;

Changing dynamic to var lets type inference take over so that db is typed as a Database instead of dynamic. Now we try to compile again:

Could not find an implementation of the query pattern for source type 'Query'.

This is a problem we can solve. The compiler translates code written using the query syntax:

var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;

Into this:

var q = db.Products.Where(p => p.UnitsInStock < 20);

Bart de Smet has a great explanation on how this works. What this means is that we can implement the Select, Where, SelectMany or any other methods on our dynamic object that will make the compiler happy.

Given that, we can implement a dummy Where method to try to get this code to compile:

class Query { 
    public dynamic Where(Expression<Func<dynamic, dynamic>> predicate) {
        return null;
    }
}
Turns out that trying to compile this yields yet another error:
An expression tree may not contain a dynamic operation
Three different compiler errors, all to do with LINQ and dynamic. Looks like the compiler team went out of their way to block this. Let’s see if removing the 
expression works:
class Query {
    public dynamic Where(Func<dynamic, dynamic> predicate) {
        return null;
    }
}

Finally we’ve gotten the code to compile, now what? Well if this was LINQ to SQL/EF/SharePoint/etc., the query would be translated into an expression tree and executed on the server when we enumerated the results. I’m more interested in the first part of that, i.e. can we build an expression tree using dynamic LINQ.

Normally when you declare a lambda it can act like an expression tree or a delegate:

Expression<Func<int, int, int>> expr = (a, b) => a + b;
Func<int, int, int> add = (a, b) => a + b;
Console.WriteLine(expr);
Console.WriteLine(add);
The above code prints out (a, b) => (a + b) and System.Func`3[System.Int32,System.Int32,System.Int32], as expected. The compiler does all the hard work of figuring 
out how to build an expression tree from the code the user writes, but we don’t have this luxury since we’re in the dynamic world.

Enter Dynamic Expression

NOTE: If you aren’t familiar with the new dynamic in C#/VB feature, read up on it here.

We introduce a new class called DynamicExpression that we will use to make the compiler do our bidding:

class DynamicExpression : DynamicObject {
   
}
Before we dig into the implementation of this class, we have to figure out what kind of expressions we are going to build. How can we represent p.UnitsInStock in an
expression tree. Normally that would be a PropertyExpression that represents the UnitsInStock property on the Product class generated by either LINQ to SQL or

LINQ to Entities. We can’t do this since we don’t have any strong types.
For simplicity, we’re going to assume that all of the properties are integers, and that property access will be represented by a call to a Property method defined 
on DynamicExpression i.e. p.UnitsInStock will become a call to Property(“UnitsInStock”).
Now that we have that our of the way, we can get to the implemention. Lets start with TryGetMember:
class DynamicExpression : DynamicObject {
    public DynamicExpression(Expression expression) {
        GeneratedExpression = expression;
    }

    public Expression GeneratedExpression { get; private set; }

    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        MethodInfo propertyMethod = typeof(DynamicExpression).GetMethod("Property", BindingFlags.NonPublic | BindingFlags.Static);
        var expression = Expression.Call(propertyMethod, Expression.Constant(binder.Name));
        result = new DynamicExpression(expression);
        return true;
    }

    private static int Property(string propertyName) {
        return 0;
    }
}
We’re storing the generated expression in a property so we can grab it after doing a bunch of manipulation. The Property method doesn’t actually have to do anything since 
its sole purpose is to represent property access.
Next we move on to simple binary expressions:
public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
    DynamicExpression dynamicExpression = arg as DynamicExpression;
    Expression rhs = null;
    if (dynamicExpression != null) {
        rhs = dynamicExpression.GeneratedExpression;
    }
    else {
        rhs = Expression.Constant(arg);
    }
    var expression = Expression.MakeBinary(binder.Operation, GeneratedExpression, rhs);
    result = new DynamicExpression(expression);
    return true;
}
First we are checking to see if the right hand side of the expression is itself a DynamicExpression and unwrapping the underlying expression if that is the case. Otherwise it
takes the right hand side to be a constant.
Some people may have already put the big picture together, if you haven’t as yet don’t worry, it isn’t exactly obvious how this works. We are trying to create an expression
tree using dynamic dispatch, that is, using the execution of the dynamic operations to build an expression tree.

Putting the pieces together

Remember how the compiler complained earlier when we tried to use dynamic in an expression tree? That’s why our overload of Where doesn’t take a Expression<Func<dynamic, dynamic>> but instead it takes a Func<dynamic, dynamic>. We’re going to fill in that blank implementation of the Where method we left off earlier.

class Query {
    public Query(Expression expression) {
        GeneratedExpression = expression;
    }

    public Expression GeneratedExpression { get; private set; }

    public Query Where(Func<dynamic, dynamic> predicate) {
        DynamicExpression expr = predicate(new DynamicExpression(GeneratedExpression));
        return new Query(expr.GeneratedExpression);
    }
}
Let’s we go back to the original query:
dynamic db = Database.Open("name");
var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;
The compiler is going to turn this into a call to db.Products.Where(p => p.UnitsInStock < 20). When this calls our implementation of Where, we construct a DynamicExpression 
and execute the lambda (p.UnitsInStock < 20) on it. While executing the lambda, the compiler sees a member access(p.UnitsInStock) on a dynamic object p and calls our
implementation of TryGetMember. Instead of TryGetMember returning a value for p.UnitsInStock, it returns an Expression that represents a member access and wraps
it in another DynamicExpression so that the process continues. The compiler then sees the less than operator and calls our implementation of TryBinaryOperation. We use the
same trick for this as we did for member access (return an expression instead of a value). This could be done for all of the operators but I chose to two for brevity.
Now that we’ve done this, we can print out the generated expression. Here is the full program:
using System;
using System.Dynamic;
using System.Linq.Expressions;
using System.Reflection;

class Program {
    static void Main(string[] args) {
        var db = Database.Open("name");
        var q = from p in db.Products
                where p.UnitsInStock < p.Price + 1
                select p;

        Console.WriteLine(q.GeneratedExpression);
    }
}

class Database {
    private string _databaseName;
    public Database(string databaseName) {
        _databaseName = databaseName;
    }

    public static Database Open(string name) {
        return new Database(name);
    }

    public Query Products {
        get {
            return new Query(null);
        }
    }
}

class Query {
    public Query(Expression expression) {
        GeneratedExpression = expression;           
    }

    public Expression GeneratedExpression { get; private set; }

    public Query Where(Func<dynamic, dynamic> predicate) {
        DynamicExpression expr = predicate(new DynamicExpression(GeneratedExpression));
        return new Query(expr.GeneratedExpression);
    }
}

class DynamicExpression : DynamicObject {
    public DynamicExpression(Expression expression) {
        GeneratedExpression = expression;
    }

    public Expression GeneratedExpression { get; private set; }

    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        MethodInfo propertyMethod = typeof(DynamicExpression).GetMethod("Property", BindingFlags.NonPublic | BindingFlags.Static);
        var expression = Expression.Call(propertyMethod, Expression.Constant(binder.Name));
        result = new DynamicExpression(expression);
        return true;
    }

    public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
        DynamicExpression dynamicExpression = arg as DynamicExpression;
        Expression rhs = null;
        if (dynamicExpression != null) {
            rhs = dynamicExpression.GeneratedExpression;
        }
        else {
            rhs = Expression.Constant(arg);
        }
        var expression = Expression.MakeBinary(binder.Operation, GeneratedExpression, rhs);
        result = new DynamicExpression(expression);
        return true;
    }

    public override bool TryUnaryOperation(UnaryOperationBinder binder, out object result) {
        if (binder.Operation == ExpressionType.IsFalse || binder.Operation == ExpressionType.IsTrue) {
            result = false;
            return true;
        }
        return base.TryUnaryOperation(binder, out result);
    }

    private static int Property(string propertyName) {
        return 0;
    }
}

Caveats

So we were able to get some expression tree support while combining LINQ and dynamic, but there are things to be aware of.

If the lhs of any operation isn’t dynamic then our TryBinaryOperation will never get called:

var q = from p in db.Products
        where 1 > p.UnitsInStock
        select p;

This would fail since 1 isn’t a dynamic object, we need something to bootstrap the dynamic dispatch.  Exercise for the reader, can we get around this limitation?

Nested queries won’t work without casting:

var q = from c in db.Categories
        from p in c.Products
        where p.UnitsInStock < 20
        select p;
This doesn’t work right now because we didn’t implement SelectMany, but also because of the first compiler error we encountered (Query expressions over source 
type 'dynamic' are not allowed). The nested query (from p in c.Products) would be trying to query over a dynamic source.
Query syntax doesn’t work in VB. VB does semantic translation instead of syntactic translation:
Dim q = From p In db.Products
        Where p.UnitsInStock < 20
        Select p
When the VB compiler generates the method for p.UnitsInStock < 20, it injects a type check for a boolean result which completely breaks this approach. 
You can however still use the lambda syntax to work around this.
There are probably other quirks that come with using this approach, and it is clear that the C# team took time to block certain scenarios, so don’t expect this to 
be a 100% solution.
We’ve successfully abused the new dynamic feature to get it working with LINQ. Hopefully in some future version of C# and VB we get deeper integration with 
LINQ and dynamic.
Posted by davidfowl with 3 comment(s)
Filed under: , ,

Microsoft.Data.dll - A re-introduction

Before we look at the reasons for the existence of Microsoft.Data, we need some background on the WebMatrix initiative. Here are some excellent blog posts that describe the type of audience we are going after and the goals of the project:

http://weblogs.asp.net/scottgu/archive/2010/07/06/introducing-webmatrix.aspx

http://blogs.msdn.com/b/davidebb/archive/2010/07/07/how-webmatrix-razor-asp-net-web-pages-and-mvc-fit-together.aspx

http://geekswithblogs.net/mbcrump/archive/2010/07/06/the-forrest-gump-guide-to-the-new-webmatrix.aspx

These blog posts point out that WebMatrix targets a different audience than our traditional pro developer audience. These are people who are not yet developers (if they ever plan to be). They may never want to become a pro developer, but want to be able to put together a simple website. If you've internalized best practices and patterns and know your way around an ORM, then you will most likely never use this library. We are trying to attract a different kind of customer: people who are on other platforms and are already using SQL today in their web pages, and people that are trying to get into programming and haven’t chosen a platform as yet.

Most likely, if you're even reading this, you're not the intended audience, though we are still interested in your feedback.

Why Microsoft Data?

Scott Gu quote:

We've debated the role of ORMs for entry-level devs quite a bit (and have a dynamic ORM implemented - but which we didn't ship with today's preview).  For someone who understand objects and classes they can be great. When we brought in web developers to use the product for a few days, though, one interesting data point that came up repeatedly was that about 80% of them had never even heard the term "ORM".  We found it took awhile for them to conceptually grok it - whereas adhoc SQL came much easier to them. It is something we are going to continue to work on finding the right balance on.

I could not put this any better. We didn’t immediately decide to go with SQL, but after going through several options, we decided it was the simplest way to go.

Absolute beginners won’t have Visual Studio, and most of our existing data stack relies heavily on tooling support (designers, and code generators). We wanted to build something that felt like it could be authored from notepad.

Who Is it For?

Nikhil has a good blog post on personas http://www.nikhilk.net/Personas.aspx:

Mort, the opportunistic developer, likes to create quick-working solutions for immediate problems and focuses on productivity and learn as needed. Elvis, the pragmatic programmer, likes to create long-lasting solutions addressing the problem domain, and learn while working on the solution. Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learn in advance before working on the solution.

WebMatrix is trying to target beginners (like Mort, or pre-Mort). By beginners I mean, people that:

  • May not understand generics (or don’t care for them)
  • Do not care about TDD, DRY, SRP, (insert acronym here)
  • Do not understand lambdas or expression trees.
  • May not have any knowledge of OOP

I could go on and on, but the point I am trying to get across is that if any of this sounds absurd to you or if it offends you in any way as a developer, then you are probably not the target audience.

This post can be summarized by the following quote from Scott Gu’s blog:

If you are a professional developer who uses VS today then WebMatrix is not really aimed at you - at least not for your "day job".  You might find WebMatrix useful for quickly putting a blog on the web or doing lightweight scripting on the side.  But it isn't intended or focused on hard-core professional or enterprise development.  It is instead aimed more for people looking to learn how to program and/or who want to get a site up and running on the web without having to write much code.

Posted by davidfowl with 19 comment(s)
More Posts Next page »