Unhandled Exception

SignalR 1.1 beta

davidfowl — Fri, 12 Apr 2013 05:54:43 GMT

We released SignalR 1.1 yesterday with a bunch of performance improvements and new features (detailed in the release notes). Among these new features, were improvements to the .NET client including the ability to set custom headers, client certificates and a custom JSON serializer. We also fixed a long standing issue people were experiencing in IE7 and IE8 with the loading bar showing up forever in the browser. However, most of this release focused on rewriting SignalR’s scaling out story from the ground up.

Scaling out

The previous releases of SignalR’s scale out implementation did not handle a lot of scenarios when it came to scaling it across web servers. We took a step back and redesigned the core scale out architecture and backplanes for SQL Server, Azure Service Bus and Redis.

There are now 3 new packages on NuGet that you can choose from when scaling out SignalR:

These backplanes share a common base class (ScaleoutMessageBus) that contains the core logic allowing SignalR to work across a webfarm.

Using Scale out

Registering a backplane hasn’t changed much from the previous versions but there are some new features to be aware of.

ALL backplanes

Scale out backplanes have 2 modes controlled by a single flag RetryOnError. Users can create a configuration object for their backplane and specify which mode they want to use. This flag determines if SignalR should start buffering messages when the scale out backplane encounters errors; by default this flag is false. Here is an example:

In the above scenario, if the Redis service went offline, with RetryOnError = false the call to send a message to all clients would throw synchronously. If RetryOnError was true, the OnError callback would be raised the call would be queued until Redis came back online. This queue does have a limit and when it is full it will also fail.

SQL SERVER

The SQL backplane requires that you create the database before hand.
The SQL scale out backplane works with or without service broker turned on.
Performance is usually better with service broker on since this enables query notifications to work.

Once the database is set up, you can enable SQL like this:

It’s all you need to get running with SQL server.

SERVICE BUS

The new service bus scale out backplane only works with the Windows Azure service. The wiki has been updated with new steps that show how to setup service bus on Windows Azure.

Windows Azure Websites is also supported.

TRACING

If you are encountering problems with scale out and want to troubleshoot what’s going on, you can turn on tracing with this xml snippet in your config file:

The purpose of this beta release is to get feedback, so do give it a try and let us know what you think. As always, file issues on github, and questions on stackoverflow, jabbr or the SignalR forum.

Microsoft ASP.NET SignalR

davidfowl — Mon, 12 Nov 2012 02:18:42 GMT

Yes, you read that right, Microsoft.AspNet.SignalR. It’s been a while since I last blogged as we’re super busy trying to make SignalR rock solid and 1.0 quality for you all to use. I’ve also given my first talk ever at Monkeyspace and then BUILD - both on SignalR. SignalR 1.0 alpha has been live for just over 2 weeks on NuGet, and a few seconds ago I just published alpha2. A lot has changed since 0.5.3 and there have been several people poking around trying to find out what’s changed and what’s new in the alpha. There’s also been lots of confusion about what to use, so I’m going to clear things up by enumerating the list of major changes.

New Packages

There are new packages that you can use today in your applications.

Microsoft.AspNet.SignalR – meta package (use this)
Microsoft.AspNet.SignalR.Client – .NET 4 and WinRT client
Microsoft.AspNet.SignalR.JS – The Javascript client.
Microsoft.AspNet.SignalR.Core – Core server package with no host implementation
Microsoft.AspNet.SignalR.Hosting.AspNet – The ASP.NET host
Microsoft.AspNet.SignalR.Hosting.Utils – utilities for signalr (signalr.exe)
Microsoft.AspNet.SignalR.Redis – Redis message bus implementation
Microsoft.AspNet.SignalR.ServiceBus – Service bus message bus implementation

If you’re using the NuGet dialog, pick the IncludePrerelease in the dropdown to see them. If you’re using the package manager console, use the –pre flag to install it.

Install-Package Microsoft.AspNet.SignalR –pre

Visual Studio Integration

When you install the ASP.NET 2012 Fall Update, you’ll see a few new things in your Add New Item menu.

These will bring in the alpha1 packages but you can update them using NuGet to get the latest and greatest.

Hub API changes

First thing to notice is the client and server members on the hub object. This came about because there was a subtle issue with naming methods on the client and server. If they were called the same thing it just wouldn’t work so we decided to separate the namespaces. As you already guessed, you can declare methods that the server calls on the client object and call methods that exist on the server with the server object.

On the server side we’ve added quite a few new APIs for hubs after lots of feedback and discussion.

This is a dump of all the new surface we’ve added for hubs. The comments are pretty self explanatory but as part of this work we resolved this long standing feature request https://github.com/SignalR/SignalR/issues/105 (it’s almost a year old!).

Connected, Disconnected and Reconnected

Before we had these awful interfaces you had to implement to detect this, now it’s just an override on the Hub itself.

No more magic?

There’s obviously still some magic, just a bit less than before. When using hubs, people always wondered how this magic ~/signalr/hubs url was getting registered. This is now an explicit call the developer has to make to enable this route. When you install the new package, it’ll add a RegisterHubs.cs file to the project in the App_Start folder.

There's even a little comment telling you what MapHubs does. If you want to, you can remove this file and just add the call to your global asax with the rest of your route registration.

Hub Pipeline

This is the most powerful feature we’ve added in the new release. Think of the hub pipeline like mvc filters but more powerful. You get to inject any piece of code anywhere in the hub pipeline for incoming and outgoing calls. Here’s an example of a hub pipeline module that logs all outgoing and incoming calls.

The module has a bunch of different methods you can override, the above sample only shows 2 of them. Each method takes the appropriate context so you can get whatever information you want about the incoming or outgoing invocation. The next feature on the list builds on the pipeline implementation.

Authorization

Many of you wanted an easy way to mark a hub as requiring authorization and now you can do that by just specifying an attribute similar to MVC.

Now you can decorate your hubs or hub methods or both to get different granularities of security. The first example shows requires authorization for all connections to the hub and calls to any incoming methods. In the second example, any connection to the hub as well as calls to Send require authorization, but calls to Send2 do not.

Client side keep alive

This feature allows clients to react to slow/buggy networks that drop connectivity often. The JavaScript client checks for a keep alive ping from the server on a configurable interval and raises the “connection slow” event before it drops the connection and starts reconnecting.

Rejoining Groups

There’s tons of confusion about how groups work in SignalR and how it works in scaled out environments (http://stackoverflow.com/questions/11143220/signalr-groups-filtering-handled-on-client-or-server). SignalR stores enough state on the server to make groups work, but all groups that a client is a member of is round-tripped with that client. This of course means that any client can claim they are in a group and spy on your messages. Groups are not a security feature, it’s about message filtering. By default, SignalR will ignore groups that are sent from client to server (to be secure by default). Clients will not be able to receive messages for groups they were in after a reconnect for most transports, but for longpolling it will be immediate. Developers can re-enable the default behavior by enabling a pipeline module that trusts the client’s groups.

If you’re more advanced and want to apply a custom filter to pre-process groups for particular clients, you can write a custom pipeline module to do so.

New Scale Out Providers

We’ve completely re-worked the scale out providers for service bus and redis and added sql server to the list (not shipping with this release but you can use the source). As part of this work, we’ve abstracted a ScaleOutMessageBus base class that each of these providers derive from in order to cleanly plug into SignalR. Check out the class for more details.

More Memory Efficient and huge throughput gains

The MessageBus was complete re-written and optimized for small messages (we normally test with 32 byte sized message payloads) and high throughput. More memory efficient means that we can get more connections on a single node. By the time we RTM, we’ll have documentation and guidance on how to do capacity planning on your webservers and on azure.

Performance Counters

We have a utility called signalr.exe (in the Microsoft.AspNet.SignalR.Hosting.Utils package) that you can use to do a few things.

Install/Uninstall SignalR performance counters
Generate a static js file for the magic ~/signalr/hubs url.

You can install the performance counters if you want to see more detailed information about your SignalR applications. Here’s a view of what that looks like:

NOTE: To use these on full IIS you need to add the app pool user for your application to the Performance Monitor Users group.

Message payload size

We’ve made some changes (not in alpha2 but later) to reduce the size of our payload dramatically and as a result changed the protocol for 1.0 as well. I’ll talk about this more in a later post.

Documentation

We’re in the process of updating it as well as preparing content for http://www.asp.net/signalr. Stay tuned!

THANK YOU ALL

Thank you all for being a part of this amazing journey from side project to official project. Thanks to your enthusiasm it has has grown rapidly into a framework that people just love to use! Keep giving us feedback! Ping me on jabbr or twitter if you have any questions!

SignalR 0.5.1 Released

davidfowl — Mon, 11 Jun 2012 05:17:01 GMT

Here comes another exciting release of SignalR. This release has some new features that you can take advantage of as well as a few bug fixes. Let’s go over the big changes for this release.

Windows 8 Client support

With the Windows 8 Release Preview, we’ve finally added a new client to support “.NET for Metro Style Apps” and also made some fixes so that SignalR’s JavaScript client works with WinJS.

WebSocket Support

Previously, websocket support required a separate package, now it’s built into SignalR. The ASP.NET host is the only one that supports this and that’s only when running an application targeting .NET 4.5 on IIS8.

Improved Cross Domain Support

We introduced cross domain support for SignalR in 0.5. In this release we’ve improved that support in a few ways:

All SignalR end points are CORS enabled by default. This means you can point the JavaScript client to the endpoint and have it just work without having to manually add the cors header to web.config.
SignalR will choose automatically negotiate the transport so there’s no need to specify it or the xdomain flag (which is no longer supported).

$.connection.hub.url = 'http://localhost:8081/signalr;
 
$.connection.hub.start();

The above code is all you need to do to connect to a remote server using SignalR.

Connection State Changes

A SignalR connection has always been a black box to the user and this made it hard to detect changes in the underlying connection state. We’ve exposed a new stateChanged event in the JavaScript and .NET clients. This allows you to listen for state changes and react to them in different ways. Here’s an example that shows the user a message if the connection went into a reconnecting state and didn’t recover after 10 seconds:

var chat = $.connection.chat;
var timeout = null;
var interval = 10000;
 
chat.addMessage = function (msg) {
    $('#messages').append('<li>' + msg + '</li>');
};
 
$.connection.hub.stateChanged(function (change) {
    if (change.newState === $.signalR.connectionState.reconnecting) {
        timeout = setTimeout(function () {
            $('#state').css('backgroundColor', 'red')
                        .html('The server is unreachable...');
        }, interval);
    }
    else if (timeout && change.newState === $.signalR.connectionState.connected) {
        $('#state').css('backgroundColor', 'green')
                        .html('The server is online');
 
        clearTimeout(timeout);
        timeout = null;
    } 
});
 
$.connection.hub.start();

When the connection state changes to reconnecting, we set a 10 second timer, if the state doesn’t change to “connected” in those 10 seconds then we set the message to say the server is unreachable. When the connection comes back online (i.e the state the connected), we set the message, “The server is online”.

This feature should give app developers a chance to deliver nicer user experiences when for clients on crappy connections or when servers go down unexpectedly. We’ll continue to improve this in the future.

Clean Disconnect

Normally when a connection is broken between the browser and server, the underlying http request takes some time to go away (you can use fiddler to see this). In 0.5.1 we’ve added the ability for the client to send a packet to the server whenever a connection is stopped or the browser is closed. This means disconnect will fire more immediately on browser close or on page refreshes.

Hubify.exe

It’s not a nuget package but will be in the http://chocolatey.org/ gallery soon. For now, you can run get hubify.exe by building https://github.com/SignalR/SignalR/tree/master/SignalR.ProxyGenerator. You can run this against your project’s bin directory or a remote url to generate or download the hubs.js file that would normally be dynamically generated when referencing the magic script url:

<script src="signalR/hubs" type="text/javascript"></script>

You can add this to a post build event in your project to generate the file a build time so that you can take advantage of http caching when you deploy your application (See an example here).

Keep Alive

Keep alive is now on by default and is set to 30 seconds.

Those are the highlights, view the release notes for more information.

Get it on NuGet today!

API Improvements made in SignalR 0.5

davidfowl — Fri, 04 May 2012 21:23:32 GMT

We made lots of breaking changes in 0.5 in order to gain more API consistency across the board. Some of the changes may not affect you if you weren’t doing anything with lower level APIs. These types of breaking changes will obviously slow down as we get closer to a 1.0 release.

API comments

This release, we spent some time writing API comments that should be helpful during development.

Breaking changes

A bunch of APIs were removed from PersistentConnection including the following:

All overloads of Send
SendToGroup
AddToGroup
RemoveFromGroup

Sending Data

All data is now sent through an IConnection (Connection property). Now, whether you’re within a PersistentConnection or outside of one, the logic to send data over that connection will look exactly the same.

SignalR 0.4

public class MyConnection : PersistentConnection
{
    protected override Task OnConnectedAsync(IRequest request, IEnumerable<string> groups, string connectionId)
    {
        return Connection.Broadcast("A new connection was made!");
    }
 
    protected override Task OnReceivedAsync(string connectionId, string data)
    {
        return Send(data);
    }
}

SignalR 0.5

public class MyConnection : PersistentConnection
{
    protected override Task OnConnectedAsync(IRequest request, string connectionId)
    {
        return Connection.Broadcast("A new connection was made!");
    }
 
    protected override Task OnReceivedAsync(string connectionId, string data)
    {
        return Connection.Send(connectionId, data);
    }
}

Groups

Group management is now centralized and consistent across Hubs and PersistentConnections. Each of those types have a property Groups of type IGroupManager that are scoped to a specific PersistentConnection or Hub. Here’s an example of adding a group in 0.4 and 0.5:

SignalR 0.4

public class MyConnection : PersistentConnection
{
    protected override Task OnConnectedAsync(IRequest request, IEnumerable<string> groups, string connectionId)
    {
        return AddToGroup(connectionId, "foo");
    }
 
    protected override Task OnReceivedAsync(string connectionId, string data)
    {
        return SendToGroup("foo", data);
    }
}

public class MyHub : Hub
{
    public Task Join(string group)
    {
        return AddToGroup(group);
    }
 
    public Task Send(string group, string message)
    {
        return Clients[group].addMessage(message);
    }
}

SignalR 0.5

public class MyConnection : PersistentConnection
{
    protected override Task OnConnectedAsync(IRequest request, string connectionId)
    {
        return Groups.Add(connectionId, "foo");
    }
 
    protected override Task OnReceivedAsync(string connectionId, string data)
    {
        return Groups.Send("foo", data);
    }
}

public class MyHub : Hub
{
    public Task Join(string group)
    {
        return Groups.Add(Context.ConnectionId, group);
    }
 
    public Task Send(string group, string message)
    {
        return Clients[group].addMessage(message);
    }
}

Dependency Injection

We’ve removed the AspNetHost type that from SignaR 0.4 and added a type that is purely optional to use, GlobalHost. The idea behind this type is to allow easy access to items in the container that will be used most commonly in any SignalR application, as well as a host agnostic default dependency resolver. This type exists in a new assembly SignalR.Hosting.Common.dll.

Replacing the resolver

SignalR 0.4

public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        AspNetHost.SetResolver(new CustomResolver());
    }
}

SignalR 0.5

public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        GlobalHost.DependencyResolver = new CustomResolver();
 
        RouteTable.Routes.MapHubs();
    }
}

Here, we’re changing the global resolver and remapping the default hub route to pick it up. The hub route is automatically added at a very early stage in the application (PreApplicationStart) so we need to remap it to use the new default resolver or we can specify a new one for the hub route directly:

public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        RouteTable.Routes.MapHubs(new CustomResolver());
    }
}

The above logic doesn’t replace the global resolver but instead creates a self contained SignalR world using the specified resolver. If you go with this more “pure” approach, you’re opting out of using the GlobalHost.

Changing the Hub route

In 0.4, the magical signalr/hubs url was implemented as a dynamic http module. Now it’s just a regular route in the route table that can be replaced. Here’s an example of changing it to signalr2:

public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        RouteTable.Routes.MapHubs("~/signalr2");
    }
}

Obviously this change needs to be accompanied by a change to your JavaScript include:

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <script src="Scripts/jquery-1.6.4.js" type="text/javascript"></script>
    <script src="Scripts/jquery.signalR-0.5rc.js" type="text/javascript"></script>
    <script src="signalr2/hubs" type="text/javascript"></script>
</head>
<body>
 
</body>
</html>

Getting a reference to the connection/hub outside of a connection or hub

This is an extremely popular pattern for integrating with other frameworks. Here’s an example of broadcasting to all clients from an mvc controller:

SignalR 0.4

public class HomeController : Controller
{
    public void PerformLongRunningOperation()
    {
        var connection = AspNetHost.DependencyResolver.Resolve<IConnectionManager>().GetConnection<MyConnection>();
 
        connection.Broadcast("Called from an mvc controller");
    }
 
    public void PerformLongRunningHubOperation()
    {
        var clients = AspNetHost.DependencyResolver.Resolve<IConnectionManager>().GetClients<MyHub>();
 
        clients.notify("Hello world");
    }
}

SignalR 0.5

public class HomeController : Controller
{
    public void PerformLongRunningOperation()
    {
        IPersistentConnectionContext context = GlobalHost.ConnectionManager.GetConnectionContext<MyConnection>();
 
        context.Connection.Broadcast("Called from an mvc controller");
    }
 
    public void PerformLongRunningHubOperation()
    {
        IHubContext context = GlobalHost.ConnectionManager.GetHubContext<MyHub>();
 
        context.Clients.notify("Hello world");
    }
}

With this, you can inject these new context objects into your mvc controller or whatever framework you want to combine with SignalR. For example:

public class HomeController : Controller
{
    private readonly IHubContext _hubContext;
 
    public HomeController(IHubContext hubContext)
    {
        _hubContext = hubContext;
    }
        
    public void PerformLongRunningHubOperation()
    {
        _hubContext.Clients.notify("Hello world");
    }
}

Configuring SignalR

On each host (Self host and GlobalHost) there’s a Configuration property. This gives you access to the server configuration for things like setting timeouts, the keep alive interval and the heart beat interval:

public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        GlobalHost.Configuration.KeepAlive = TimeSpan.FromSeconds(20);
    }
}

I’ll be blogging about more of these changes as we get closer to the release of 0.5.

SignalR 0.5

davidfowl — Thu, 03 May 2012 04:29:38 GMT

This is the rebirth of my blog. I’ve been busy over the last few months with projects like JabbR, SignalR and other stuff. With the upcoming release of SignalR 0.5, I thought it’d be good to write a blog post that touches on some of the things we did.

Webfarm Support

There’s a lot of confusion as to what webfarm support means in SignalR. This is an FYI, SignalR 0.5 will not automatically make your application work on azure with zero effort. SignalR needs a way to communicate between instances of your webserver, in 0.5 we implemented 2 of those providers on NuGet (message buses):

SignalR.WindowsAzureServiceBus - http://vasters.com/clemensv/2012/02/13/SignalR+Powered+By+Service+Bus.aspx (blog post slightly outdated but information is relevant).
SignalR.Redis – Redis message bus implementation for SignalR - https://github.com/SignalR/SignalR.Redis

When deciding to make SignalR work in a webfarm, you’ll have these 2 options. We’re going to continue improving them as well as implement solutions on top of popular message buses like NServiceBus and lower fidelity solutions built on top of SQL QNS and possibly SQL Server Service Broker.

This wasn’t the only hurdle to making SignalR work in webfarms, there were some nasty bugs that caused some bad behavior that we fixed:

Disconnect notifications across webfarms
Load balancer idle timeout in azure.

Fixing disconnect in webfarms was the most important issue to fix. Before SignalR 0.5rc, disconnect in a webfarm (using service bus or redis) could fire for the same client multiple times. This is more obvious when using longPolling and load balancing happens between web server instances. If you had control of your LB, you could affinitize clients to a server so you’d never see this behavior, but I’m glad to say we got this fixed.

For load balancer idle timeouts (especially on azure), we support keep-alive. You can now tell SignalR to send an empty packet on a specific time interval. This will fend off nosy network devices that like to kill idle connections waiting for data. This is especially important for things like server sent events and forever frame transports.

Mono Support

In the 0.4 release, SignalR only “worked” when self hosted on mono so I spent time making SignalR work across the board (ASP.NET and all). When I say work, I mean both the client and the server. You can now literally clone SignalR onto your OSX (I didn’t try linux) box and run make and get a successful build (see the README for more details). This wasn’t a simple task as I had to make changes to the NuGet targets file so that package restore worked on mono.

If you’re using Mono develop, open SignalR.Mono.sln. You can run the samples as you would normally. Here’s a glimpse of the chat demo running:

Not much time was spent optimizing and tuning specifically for mono but in the future we’ll have time to do that.

JsonP Longpolling support (XDomain)

This feature wasn’t planned for 0.5 but thanks to an awesome contribution from codeputty, this became a reality.

Dynamic Hubs Implementation

Contributor pszmyd was trying to build better integration between SignalR and Orchard and ended up adding infrastructure to SignalR to decouple Hubs from types (System.Type). This is a great example of why I love OSS development (scratch your own itch).

Continuous Integration and NuGet Server

We finally got a CI server setup on codebetter’s teamcity instance. You can view SignalR here http://teamcity.codebetter.com/project.html?projectId=project188. The CI also builds packages and pushes them to our myget feed. If you want to live on the bleeding edge but don’t want to build from source, you can configure nuget to point to http://www.myget.org/F/signalr/ (make sure you use the –Pre flag or select the Include Prerelease in the dialog).

0.5 Release

Help us test the RC now on NuGet! (http://signalr.net/releases/v0584.html). We’re reserving this week for any bugs that we consider ship stoppers.

What’s Next

We’re going to continue improving SignalR. Future releases will focus on some of the following:

Performance Improvements
In the box WebSockets support for Windows8
Windows 8 .NET Client
.NET Micro Framework Client
Web API Host
Build time hubs.js generation (instead of runtime).
Better network availability handling
Better Mobile support (optimized for battery life)
Web Forms control

… And much more! Check the issue backlog for details on what else we've got planned for upcoming releases.

Come Talk with us

We’re always around in the signalr room on jabbr. Come say hi!

Razor Debugger for ASP.NET WebPages

davidfowl — Mon, 15 Nov 2010 07:47:00 GMT

I have neglected blogging recently because I’ve been super busy working on Razor tooling, NuGet and Beta 3 Release of the Web Stack. Now that I have some free time, I can blog about a really cool project I’ve been working on.

The Razor Debugger is a simple web based debugger for ASP.NET WebPages (it might work in MVC but hasn’t really be tested). It started off as an experiment to see if it was possible to have a simple debugging experience for developers using WebMatrix.

It’s a NuGet Package

Of course the Razor Debugger is distributed as a NuGet package and is available today on the WebMatrix feed. You can learn more about ASP.NET WebPages package management here.

Debugging Tutorial

Let’s step through a simple example on how to use the debugger with your site.

Start by opening WebMatrix and choose Site From Template. Create a new Empty Site and create a new cshtml page.

Next, we’re going to install the RazorDebugger through the admin UI. Launch the website and navigate to the _Admin url in your browser:

Type your secret password and continue to the Package Manager. List all packages from the online feed and install the RazorDebugger package:

After installing the debugger, navigate to the debugger page by going to ~/debugger.

The debugger is a section of the website that shows all of the pages on the left, and some UI on the right that we’ll describe in further details below.

To start debugging your site, put some code in the page and click on the page in the debug view.

Now that we have some code in our page we can set a break point by clicking on the gray pane beside the line numbers in source view. When a break point is set, a red circle will be placed on the line where the break point was set.

Start debugging! Click on Launch Page at the top of the page to run the page. Switch back to the debugger and see the break point being hit.

Now that we’re debugging, we can use the Step Into, Step Over, Step Out and Run buttons or the associated shortcut keys. Also, as you step, the locals, watches and stack trace UI at the bottom will change to show locals and watches respectively.

You can also change the values of locals by clicking on the value of the local and typing a new one.

Stepping into other pages

It’s possible to step into other pages if a call to RenderPage is made within the page you are debugging. I’ve updated the sample to contain 2 pages, Page.cshtml passes an anonymous type with the Name “David” to Page2.cshtml.

Now we can hit Step Into or i and step into Page2.cshtml.

Exceptions

The debugger also supports unhandled exceptions. The page below shows an example of a NullReferenceException. When an exception occurs the line is highlighted with a dotted border (it’s kind of hard to see) and the exception object shows up in the locals window.

Start Debugging

This debugger is a prototype and isn’t meant to replace the debugger in visual studio by any means. It is however a pretty cool experiment and you can try it out today in your ASP.NET WebPages!

Dynamic LINQ Part 2 (Evolution)

davidfowl — Thu, 19 Aug 2010 08:34:00 GMT

A while ago, Scott Gu blogged about a Dynamic LINQ library that was part of the C# samples in VS2008. This library let developers put together late bound LINQ queries using a special syntax. This really comes in handy when you want to build a LINQ query based on dynamic input. Consider the following example:

public ActionResult List(string sort) {
    var northwind = new NorthwindEntities();
    IQueryable<Product> products = northwind.Products;
    switch (sort) {
        case "ProductName":
            products = products.OrderBy(p => p.ProductName);
            break;
        case "UnitsInStock":
            products = products.OrderBy(p => p.UnitsInStock);
            break;
        default:
            break;
    }
 
    return View(products.ToList());
}

In the above example, we’re trying to sort a list of products based on the sort parameter before we render the view. Writing the above code is painful, especially if you have a large number of sortable fields. Of course this isn’t the only way to write this code, in fact, if you’re a LINQ expert you can build an expression tree at runtime that represents the sort expression, but that solution is far from trivial to implement.

Last time we showed how we can (ab)use C# 4 dynamic feature to enable LINQ queries against dynamic types. Today I’ll show you how we can expand on that concept, and turn our previous solution into a general purpose library that allows us to execute late bound queries against existing LINQ providers.

The library itself is pretty small, it consists of 2 classes:

DynamicExpressionBuilder – The dynamic object that we used last time to build the actual expression tree
DynamicQueryable – a static class with a bunch of extension methods that are parallel to the actual Queryable static class

DynamicExpressionBuilder uses the same trick as last time (dynamic dispatch to build expressions), but this time we’re building expressions that would be recognized by most LINQ providers.

public override bool TryGetMember(GetMemberBinder binder, out object result) {
    result = new DynamicExpressionBuilder(Expression.Property(Expression, binder.Name));
    return true;
}
 
public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
    result = MakeBinaryBuilder(Expression, binder.Operation, arg);
    return true;
}
 
public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {            
    MethodInfo methodInfo = null;            
    if (args.Any(a => a == null)) {
        methodInfo = Expression.Type.GetMethod(binder.Name);
    }
    else {
        var types = args.Select(arg => arg.GetType()).ToArray();
        methodInfo = Expression.Type.GetMethod(binder.Name, types);
    }            
 
    if (methodInfo != null) {
        var expression = Expression.Call(Expression, methodInfo, args.Select(Expression.Constant));
        result = new DynamicExpressionBuilder(expression);
        return true;
    }
    return base.TryInvokeMember(binder, args, out result);
}

The actual implementation overrides more methods, but for brevity we’re only going to look at 3 of them. Each method captures the intent of the code 
and turns it into the appropriate expression tree.

public static class DynamicQueryable {            
    public static IQueryable Where(this IQueryable source, Func<dynamic, dynamic> predicate) {
        return Where(source, GetExpression(source.ElementType, predicate));
    }
 
    private static LambdaExpression GetExpression(Type elementType, Func<dynamic, dynamic> expressionBuilder) {
        ParameterExpression parameterExpression = Expression.Parameter(elementType, expressionBuilder.Method.GetParameters()[0].Name);
        DynamicExpressionBuilder dynamicExpression = expressionBuilder(new DynamicExpressionBuilder(parameterExpression));
        Expression body = dynamicExpression.Expression;
        return Expression.Lambda(body, parameterExpression);
    }
 
    private static IQueryable Where(IQueryable source, LambdaExpression expression) {
        return source.Provider.CreateQuery(Expression.Call(typeof(Queryable),
                                                            "Where",
                                                            new[] { source.ElementType },
                                                            source.Expression,
                                                            Expression.Quote(expression)));
    }

    …etc
}

Normally when you’re writing LINQ queries, the implementations of Where, Select, SelectMany and other extension methods come from either the Enumerable or Queryable 
class.The DynamicQueryable class uses this pattern to provide overloads of the same extension methods on the non generic IQueryable.

This lets us do a lot of cool things with existing LINQ providers. Let’s look at some examples:

using System;
using System.Linq;
 
class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);
 
        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}

Above we have a regular LINQ to Entities query against Northwind. Notice the usings are System, and System.Linq, those make sure that you’re using the Queryable 
implementations of Where and OrderBy.

We can write this same code in with Dynamic LINQ:

using System;
using DynamicLINQ;
 
class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);
 
        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}

We replaced the System.Linq using with a DynamicLINQ using, which makes use of our extensions methods. If we wanted to sort by a dynamic variable you could do it 
using the indexer syntax supported by DynamicQueryBuilder:

using System;
using DynamicLINQ;
 
class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p["ProductName"]);
 
        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}

That’s all pretty cool, but there are more scenarios that can’t be done today using regular LINQ which Dynamic LINQ makes easy.

One of the biggest annoyances for me using LINQ today, is the fact that custom functions can’t be used inside of query expressions. This makes total sense once 
you understand how LINQ expressions work, but if it were possible, you could make big queries a lot be more maintainable.

Take the following query using LINQ to Entities:

using System;
using System.Linq;
 
class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
 
        var query = northwind.Products.Where(p => IsCheap(p));
 
        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
 
    public static bool IsCheap(Product product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}

Anyone who has done any programming with LINQ to Entities knows that this won’t work. When we try to run this code, we get an error saying that this expression 
cannot be converted to SQL, or more precisely:

Unhandled Exception: System.NotSupportedException: LINQ to Entities does not recognize the method 'Boolean IsCheap(Sample.Product)' method, and this method cannot be translated into a store expression

The reason this doesn’t work is because the compiler puts the IsCheap method call in the expression tree and EF tries to convert it into SQL:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => IsCheap(p))

What we really wanted is for the compiler to replace IsCheap with the expression in the body of IsCheap, i.e. replace IsCheap with !product.Discontinued && product.UnitsInStock < 10.

This same expression would work today with DynamicLINQ:

using System;
using DynamicLINQ;
 
class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
 
        var query = northwind.Products.Where(p => IsCheap(p));
 
        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
 
    public static dynamic IsCheap(dynamic product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}

We tweaked the method slightly, changing the argument and return type to be dynamic. When we run this code it works and we see a list of cheap products.

The difference here is that we’re actually executing code to build the expression tree, so instead of putting the method call into the resultant tree we actually invoke the method and the expression tree ends up looking like this:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => (Not(p.Discontinued) And (Convert(p.UnitsInStock) < 10)))

There are also some scenarios that don’t work in the current implementation. For example, casting and some projections don’t work well with existing LINQ providers. I’ve put the code on a bitbucket repository.

http://bitbucket.org/dfowler/dynamiclinq

Enjoy!

Dynamic LINQ (A little more dynamic)

davidfowl — Wed, 04 Aug 2010 08:15:17 GMT

It seems I’ve become (in)famous in the last two days. With all the buzz around Microsoft.Data, I figure I’d post on something else that is somewhat related.

There were a bunch of comments on my previous posts that said we should do something to this effect:

dynamic db = Database.Open("name");
var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;

Which looks like LINQ but it would be over a dynamic object. Of course you don’t get the strong typing benefits of LINQ since there are no classes, but at least the 
syntax is nice and the user is protected from SQL injection.

Disclaimer

This blog post is a technical discussion about the limitations that exist today around LINQ and dynamic and interesting ways to get around it. Nothing more. Continue reading.

Unfortunately when we try to compile this the compiler screams:

Query expressions over source type 'dynamic' or with a join sequence of type 'dynamic' are not allowed

Maybe it’s better if we just use a real ORM now, or the other Dynamic LINQ library, but now I’m curious and want to figure out if this is even possible. Let’s try it another way:

dynamic db = Database.Open("name");
var q = db.Products.Where(p => p.UnitsInStock < 20);

The compiler complains again, but this time with a different error:

Cannot use a lambda expression as an argument to a dynamically dispatched operation without first casting it to a delegate or expression tree type

So we’re being blocked in two different ways. Can we work around this somehow?

Since we can’t query over a dynamic object we’ll give the Database and the Products property static types:

class Database {
    private string _databaseName;
    public Database(string databaseName) {
        _databaseName = databaseName;
    }
 
    public static Database Open(string name) {
        return new Database(name);
    }
 
    public Query Products {
        get {
            return new Query();
        }
    }
}
 
class Query {

}

Now we can do this:

var db = Database.Open("name");
var q = from p in db.Products
        where p.UnitsInStock < 20
        select p;

Changing dynamic to var lets type inference take over so that db is typed as a Database instead of dynamic. Now we try to compile again:

Could not find an implementation of the query pattern for source type 'Query'.

This is a problem we can solve. The compiler translates code written using the query syntax:

var q = from p in db.Products
where p.UnitsInStock < 20
select p;

Into this:

var q = db.Products.Where(p => p.UnitsInStock < 20);

Bart de Smet has a great explanation on how this works. What this means is that we can implement the Select, Where, SelectMany or any other methods on our dynamic object that will make the compiler happy.

Given that, we can implement a dummy Where method to try to get this code to compile:

class Query { 
    public dynamic Where(Expression<Func<dynamic, dynamic>> predicate) {
        return null;
    }
}

Turns out that trying to compile this yields yet another error:

An expression tree may not contain a dynamic operation

Three different compiler errors, all to do with LINQ and dynamic. Looks like the compiler team went out of their way to block this. Let’s see if removing the 
expression works:

class Query {
    public dynamic Where(Func<dynamic, dynamic> predicate) {
        return null;
    }
}

Finally we’ve gotten the code to compile, now what? Well if this was LINQ to SQL/EF/SharePoint/etc., the query would be translated into an expression tree and executed on the server when we enumerated the results. I’m more interested in the first part of that, i.e. can we build an expression tree using dynamic LINQ.

Normally when you declare a lambda it can act like an expression tree or a delegate:

Expression<Func<int, int, int>> expr = (a, b) => a + b;
Func<int, int, int> add = (a, b) => a + b;
Console.WriteLine(expr);
Console.WriteLine(add);

The above code prints out (a, b) => (a + b) and System.Func`3[System.Int32,System.Int32,System.Int32], as expected. The compiler does all the hard work of figuring 
out how to build an expression tree from the code the user writes, but we don’t have this luxury since we’re in the dynamic world.

Enter Dynamic Expression

NOTE: If you aren’t familiar with the new dynamic in C#/VB feature, read up on it here.

We introduce a new class called DynamicExpression that we will use to make the compiler do our bidding:

class DynamicExpression : DynamicObject {
    
}

Before we dig into the implementation of this class, we have to figure out what kind of expressions we are going to build. How can we represent p.UnitsInStock in an
expression tree. Normally that would be a PropertyExpression that represents the UnitsInStock property on the Product class generated by either LINQ to SQL or 

LINQ to Entities. We can’t do this since we don’t have any strong types.

For simplicity, we’re going to assume that all of the properties are integers, and that property access will be represented by a call to a Property method defined 
on DynamicExpression i.e. p.UnitsInStock will become a call to Property(“UnitsInStock”).

Now that we have that our of the way, we can get to the implemention. Lets start with TryGetMember:

class DynamicExpression : DynamicObject {
    public DynamicExpression(Expression expression) {
        GeneratedExpression = expression;
    }
 
    public Expression GeneratedExpression { get; private set; }
 
    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        MethodInfo propertyMethod = typeof(DynamicExpression).GetMethod("Property", BindingFlags.NonPublic | BindingFlags.Static);
        var expression = Expression.Call(propertyMethod, Expression.Constant(binder.Name));
        result = new DynamicExpression(expression);
        return true;
    }
 
    private static int Property(string propertyName) {
        return 0;
    }
}

We’re storing the generated expression in a property so we can grab it after doing a bunch of manipulation. The Property method doesn’t actually have to do anything since 
its sole purpose is to represent property access.

Next we move on to simple binary expressions:

public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
    DynamicExpression dynamicExpression = arg as DynamicExpression;
    Expression rhs = null;
    if (dynamicExpression != null) {
        rhs = dynamicExpression.GeneratedExpression;
    }
    else {
        rhs = Expression.Constant(arg);
    }
    var expression = Expression.MakeBinary(binder.Operation, GeneratedExpression, rhs);
    result = new DynamicExpression(expression);
    return true;
}

First we are checking to see if the right hand side of the expression is itself a DynamicExpression and unwrapping the underlying expression if that is the case. Otherwise it
takes the right hand side to be a constant.

Some people may have already put the big picture together, if you haven’t as yet don’t worry, it isn’t exactly obvious how this works. We are trying to create an expression
tree using dynamic dispatch, that is, using the execution of the dynamic operations to build an expression tree.

Putting the pieces together

Remember how the compiler complained earlier when we tried to use dynamic in an expression tree? That’s why our overload of Where doesn’t take a Expression<Func<dynamic, dynamic>> but instead it takes a Func<dynamic, dynamic>. We’re going to fill in that blank implementation of the Where method we left off earlier.

class Query {
    public Query(Expression expression) {
        GeneratedExpression = expression;
    }
 
    public Expression GeneratedExpression { get; private set; }
 
    public Query Where(Func<dynamic, dynamic> predicate) {
        DynamicExpression expr = predicate(new DynamicExpression(GeneratedExpression));
        return new Query(expr.GeneratedExpression);
    }
}

Let’s we go back to the original query:

dynamic db = Database.Open("name");
var q = from p in db.Products
where p.UnitsInStock < 20
select p;

The compiler is going to turn this into a call to db.Products.Where(p => p.UnitsInStock < 20). When this calls our implementation of Where, we construct a DynamicExpression 
and execute the lambda (p.UnitsInStock < 20) on it. While executing the lambda, the compiler sees a member access(p.UnitsInStock) on a dynamic object p and calls our 
implementation of TryGetMember. Instead of TryGetMember returning a value for p.UnitsInStock, it returns an Expression that represents a member access and wraps 
it in another DynamicExpression so that the process continues. The compiler then sees the less than operator and calls our implementation of TryBinaryOperation. We use the 
same trick for this as we did for member access (return an expression instead of a value). This could be done for all of the operators but I chose to two for brevity.

Now that we’ve done this, we can print out the generated expression. Here is the full program:

using System;
using System.Dynamic;
using System.Linq.Expressions;
using System.Reflection;
 
class Program {
    static void Main(string[] args) {
        var db = Database.Open("name");
        var q = from p in db.Products
                where p.UnitsInStock < p.Price + 1
                select p;
 
        Console.WriteLine(q.GeneratedExpression);
    }
}
 
class Database {
    private string _databaseName;
    public Database(string databaseName) {
        _databaseName = databaseName;
    }
 
    public static Database Open(string name) {
        return new Database(name);
    }
 
    public Query Products {
        get {
            return new Query(null);
        }
    }
}
 
class Query {
    public Query(Expression expression) {
        GeneratedExpression = expression;            
    }
 
    public Expression GeneratedExpression { get; private set; }
 
    public Query Where(Func<dynamic, dynamic> predicate) {
        DynamicExpression expr = predicate(new DynamicExpression(GeneratedExpression));
        return new Query(expr.GeneratedExpression);
    }
}
 
class DynamicExpression : DynamicObject {
    public DynamicExpression(Expression expression) {
        GeneratedExpression = expression;
    }
 
    public Expression GeneratedExpression { get; private set; }
 
    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        MethodInfo propertyMethod = typeof(DynamicExpression).GetMethod("Property", BindingFlags.NonPublic | BindingFlags.Static);
        var expression = Expression.Call(propertyMethod, Expression.Constant(binder.Name));
        result = new DynamicExpression(expression);
        return true;
    }
 
    public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
        DynamicExpression dynamicExpression = arg as DynamicExpression;
        Expression rhs = null;
        if (dynamicExpression != null) {
            rhs = dynamicExpression.GeneratedExpression;
        }
        else {
            rhs = Expression.Constant(arg);
        }
        var expression = Expression.MakeBinary(binder.Operation, GeneratedExpression, rhs);
        result = new DynamicExpression(expression);
        return true;
    }
 
    public override bool TryUnaryOperation(UnaryOperationBinder binder, out object result) {
        if (binder.Operation == ExpressionType.IsFalse || binder.Operation == ExpressionType.IsTrue) {
            result = false;
            return true;
        }
        return base.TryUnaryOperation(binder, out result);
    }
 
    private static int Property(string propertyName) {
        return 0;
    }
}

Caveats

So we were able to get some expression tree support while combining LINQ and dynamic, but there are things to be aware of.

If the lhs of any operation isn’t dynamic then our TryBinaryOperation will never get called:

var q = from p in db.Products
        where 1 > p.UnitsInStock
        select p;

This would fail since 1 isn’t a dynamic object, we need something to bootstrap the dynamic dispatch. Exercise for the reader, can we get around this limitation?

Nested queries won’t work without casting:

var q = from c in db.Categories
        from p in c.Products
        where p.UnitsInStock < 20
        select p;

This doesn’t work right now because we didn’t implement SelectMany, but also because of the first compiler error we encountered (Query expressions over source 
type 'dynamic' are not allowed). The nested query (from p in c.Products) would be trying to query over a dynamic source.

Query syntax doesn’t work in VB. VB does semantic translation instead of syntactic translation:

Dim q = From p In db.Products
        Where p.UnitsInStock < 20
        Select p

When the VB compiler generates the method for p.UnitsInStock < 20, it injects a type check for a boolean result which completely breaks this approach. 
You can however still use the lambda syntax to work around this.

There are probably other quirks that come with using this approach, and it is clear that the C# team took time to block certain scenarios, so don’t expect this to 
be a 100% solution.

We’ve successfully abused the new dynamic feature to get it working with LINQ. Hopefully in some future version of C# and VB we get deeper integration with 
LINQ and dynamic.

Microsoft.Data.dll - A re-introduction

davidfowl — Tue, 03 Aug 2010 16:27:45 GMT

Before we look at the reasons for the existence of Microsoft.Data, we need some background on the WebMatrix initiative. Here are some excellent blog posts that describe the type of audience we are going after and the goals of the project:

http://weblogs.asp.net/scottgu/archive/2010/07/06/introducing-webmatrix.aspx

http://blogs.msdn.com/b/davidebb/archive/2010/07/07/how-webmatrix-razor-asp-net-web-pages-and-mvc-fit-together.aspx

http://geekswithblogs.net/mbcrump/archive/2010/07/06/the-forrest-gump-guide-to-the-new-webmatrix.aspx

These blog posts point out that WebMatrix targets a different audience than our traditional pro developer audience. These are people who are not yet developers (if they ever plan to be). They may never want to become a pro developer, but want to be able to put together a simple website. If you've internalized best practices and patterns and know your way around an ORM, then you will most likely never use this library. We are trying to attract a different kind of customer: people who are on other platforms and are already using SQL today in their web pages, and people that are trying to get into programming and haven’t chosen a platform as yet.

Most likely, if you're even reading this, you're not the intended audience, though we are still interested in your feedback.

Why Microsoft Data?

Scott Gu quote:

We've debated the role of ORMs for entry-level devs quite a bit (and have a dynamic ORM implemented - but which we didn't ship with today's preview). For someone who understand objects and classes they can be great. When we brought in web developers to use the product for a few days, though, one interesting data point that came up repeatedly was that about 80% of them had never even heard the term "ORM". We found it took awhile for them to conceptually grok it - whereas adhoc SQL came much easier to them. It is something we are going to continue to work on finding the right balance on.

I could not put this any better. We didn’t immediately decide to go with SQL, but after going through several options, we decided it was the simplest way to go.

Absolute beginners won’t have Visual Studio, and most of our existing data stack relies heavily on tooling support (designers, and code generators). We wanted to build something that felt like it could be authored from notepad.

Who Is it For?

Nikhil has a good blog post on personas http://www.nikhilk.net/Personas.aspx:

Mort, the opportunistic developer, likes to create quick-working solutions for immediate problems and focuses on productivity and learn as needed. Elvis, the pragmatic programmer, likes to create long-lasting solutions addressing the problem domain, and learn while working on the solution. Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learn in advance before working on the solution.

WebMatrix is trying to target beginners (like Mort, or pre-Mort). By beginners I mean, people that:

May not understand generics (or don’t care for them)
Do not care about TDD, DRY, SRP, (insert acronym here)
Do not understand lambdas or expression trees.
May not have any knowledge of OOP

I could go on and on, but the point I am trying to get across is that if any of this sounds absurd to you or if it offends you in any way as a developer, then you are probably not the target audience.

This post can be summarized by the following quote from Scott Gu’s blog:

If you are a professional developer who uses VS today then WebMatrix is not really aimed at you - at least not for your "day job". You might find WebMatrix useful for quickly putting a blog on the web or doing lightweight scripting on the side. But it isn't intended or focused on hard-core professional or enterprise development. It is instead aimed more for people looking to learn how to program and/or who want to get a site up and running on the web without having to write much code.

Introduction to Microsoft.Data.dll

davidfowl — Mon, 02 Aug 2010 16:18:00 GMT

UPDATE!

Before you continue reading this blog post, check out the prequel.

I’ve been pretty busy recently working on cool features for “ASP.NET WebPages with Razor Syntax” (what a mouth full) and other things. I’ve worked on tons of stuff that I wish I could share with you, but what I can share is something that many people haven’t blogged about - Microsoft.Data.dll.

What is Microsoft.Data

It’s an awesome new assembly/namespace that contains everything you’ll ever need to access a database. In ASP.NET WebPages we wanted people to be able to access the database without having to write too many lines of code. Any developer that has used raw ADO.NET knows this pain:

using (var connection = new SqlConnection(@"Data Source=.\SQLEXPRESS;AttachDbFilename=|DataDirectory|\Northwind.mdf;
                                           Initial Catalog=|DataDirectory|\Northwind.mdf;Integrated Security=True;User Instance=True")) {
    using (var command = new SqlCommand("select * from products where UnitsInStock < 20", connection)) {
        connection.Open();
        using (SqlDataReader reader = command.ExecuteReader()) {
            while (reader.Read()) {
                Response.Write(reader["ProductName"] + " " + reader["UnitsInStock"]);
            }
        }
    }
}

Wow, that’s a lot of code compared to:

using (var db = Database.OpenFile("Northwind")) {
    foreach (var product in db.Query("select * from products where UnitsInStock < @0", 20)) {
        Response.Write(product.ProductName + " " + product.UnitsInStock);
    }
}

The user doesn’t have to learn about connection strings or how to create a command with a connection and then use a reader to get the results. Also, the above code is tied to Sql Server since we’re using specific implementations of the connection, command, and reader(SqlConnection, SqlCommand, SqlDataReader).

Compare this with code below it. We’ve reduced the amount of lines required to connect to the database, and the syntax for accessing columns is also a lot nicer, that’s because we’re taking advantage of C#’s new dynamic feature.

Why is it so much easier you ask? Well, the Database class is what you’ll be working with when accessing data. There are several methods that let you perform different kinds of queries and factory methods for connecting to the database.

Connecting to the Database

Sql Compact 4 is our main story when developing locally with web matrix, so we optimized for the “I have a database file under App_Data in my web site and I want to access it” case. The first overload we’re going to look at does exactly that and is named appropriately, Database.OpenFile.

Database.OpenFile takes either a full path or a relative path, and uses a default connection string based on the file extension in order to connect to a database. To see this in action, run the starter site template in webmatrix and add this code to the Default.cshtml:

var db = Database.OpenFile("StarterSite.sdf");
@ObjectInfo.Print(db.Connection)

The first line will create a database object with a connection pointing to the sdf file under App_Data. The second line is taking advantage of our ObjectInfo helper 
(more on this later) to show the properties of the database object.

Looking at the properties you can see that the connection state is closed, pretty weird for a method called Open to return a closed connection no? The reason is we want to delay the work as long as possible(we don’t even create the connection up front) i.e. until we actually decide to do a query.

If you look at the ConnectionString property you’ll see |DataDirectory|\StarterSite.sdf, this is one of the default connection strings I mentioned earlier. We assume relative path means within the |DataDirectory| which is “App_Data” in this case.

For simple cases OpenFile works. One of the big downsides is the fact that you are essentially hardcoding that you are using a database file, which will make it harder to migrate to sql server in the future. We have a solution for this that I’ll talk about below.

For those developers that understand connection strings and need a litte more control, then Database.OpenConnectionString might be for you. This API does exactly what you think it does, create a database object with a connection that uses the connection string specified.

var db = Database.OpenConnectionString(@"Data Source=.\SQLEXPRESS;
                                         AttachDbFilename=|DataDirectory|\Northwind.mdf;
                                         Integrated Security=True;User Instance=True");

This is nice for more control but, connection strings are things that normally are stored in a web.config so it can be changed without recompiling the application.

Last but not least is the most magical (and controversial) solution. We went back and forth about this one but put it in the beta in order to get some feedback. 
Database.Open allows the user to specify a named connection, which can be one of 3 things (this is where the magic comes in):

A database file under the data directory (App_Data) that matches the name plus one of our supported extensions (.sdf or .mdf)

A connection registered with and Database.RegisterConnectionString, Database.RegisterFile APIs

```
A connection string from web.config
```

We had lots of debate (still ongoing) internally about what the fall back order should be and if there should even be a fallback order. Today, we look for a 
connection in all the mentioned places and throw an exception if it is ambiguous (i.e. more than one) to prevent confusion. Database.Open is what we recommend since 
it allows the user to change what connection a name maps to, making a simple migration from Sql Compact 4 to Sql Server possible without any code change.

With this in mind here are some examples on how you can use Database.Open:

File based

// (Assume you have Northwind.mdf in the database)

Database.Open("Northwind")

Register API

// _start.cshtml
Database.RegisterConnectionString("MyDatabaseServer", "Data Source=192.168.1.20;Initial Catalog=MyDb")

// SomeFile.cshtml
Database.Open("MyDatabaseServer")

Configuration

// web.config
<configuration>
<connectionStrings>
    <add name="Northwind2"
        connectionString="Data Source=.\SQLEXPRESS;Integrated Security = true;Initial Catalog=Northwind"
        providerName="System.Data.SqlClient" />
</connectionStrings>
</configuration>

Database.Open("Northwind2")

I mentioned that the code in the very first example was tied to SqlServer. We internally use ADO.NET’s data providers to construct connections, so Microsoft.Data will mostly work with any database has a registered ADO.NET provider.

Caveats

Since we optimized for what we thought would be the most common scenario we set up some default values and behaviors that may be hard to debug.
One of these is the fact that the default ADO.NET provider is Sql Compact 4. To work around these limitations we added a providerName paramter to methods like OpenConnectionString and RegisterConnectionString. We also recommend that you specify the providerName attribute when defining connection strings in web.config. Following those patterns will mitigate most of the issues.

We also provide a global override so you can change what the default provider is. For example, if you wanted to make the default provider Sql Server then you can do it by adding the following piece of configuration to your website:

<configuration>  
  <appSettings>
    <add key="systemData:defaultProvider" value="System.Data.SqlClient" />
  </appSettings>
</configuration>

Today I covered connecting to the database, next time I’ll concentrate on querying. Stay tuned…

Hacking the ASP.NET Parser

davidfowl — Tue, 01 Dec 2009 09:09:00 GMT

A while ago I blogged about using external templates in an asp.net application. The problem with that was it wasn’t natively supported, so I had to create a derived SpecialRepeater in order to take advantage of the external template system.

In this latest parser hack I’ve managed to “trick” the asp.net parser into letting me declaratively set any template property to be a virtual path in order to load a user control into that template at runtime.

How it works

The ASP.NET compliation system is pretty complex. There are all kinds of extensibility points, including BuildProviders, PageParserFilters, ControlBuilders, ExpressionBuilders and the list goes on. One unknown *feature* of the parser is it’s ability to generate code from something called an InstanceDescriptor.

The parser has a special way of dealing with ITemplate properties so if we try to do this:

<asp:Repeater ID="repeater" runat="server" ItemTemplate="~/MyUserControl.ascx">
</asp:Repeater>

It fails because the there is no way to convert the string “~/MyUserControl.ascx” into an ITemplate.

The parser uses the TypeConverter attribute defined on properties it parses to aid in the conversion. Enter TypeDescriptionProvider. These complex beasts are used at the heart of all designers in Visual Studio. There are used for things like populating the property grid, and adding and removing properties dynamically, basically a general purpose metadata API (think of it as an abstraction on top of reflection).

VirtualPathTemplate

The code we are going to generate will instantiate a VirtualPathTemplate with a virtual path pointing to a user control on our site. Normally when you define a template in markup, a special type called CompiledTemplateBuilder (which points to a delegate that builds the template at runtime) is assigned to it. We want to replace a line of code that looks like this:

repeater.ItemTemplate = new CompiledTemplateBuilder(BuildTemplate);

to this

repeater.ItemTemplate = new VirtualPathTemplate("~/MyUserControl.ascx");

TypeDescriptionProvider

After overriding about 4 classes (TypeDescriptionProvider, CustomTypeDescriptor, PropertyDescriptor, and finally TypeConverter) we are able to control what happens when the parser asks, “can we convert “~/MyUserControl.ascx” to an ITemplate?”.

Here is the code for the TemplateTypeConverter:

public override bool CanConvertTo(ITypeDescriptorContext context, Type destinationType) {
    // Allow InstanceDescriptor so that the code gen engine can use it to generate the correct
    // code for the ITemplate property
    return destinationType == typeof(InstanceDescriptor) || _converter.CanConvertTo(context, destinationType);
}

public override object ConvertTo(ITypeDescriptorContext context, CultureInfo culture, object value, Type destinationType) {
    var descriptorProvider = value as IInstanceDescriptorProvider;
    if (descriptorProvider != null) {
        return descriptorProvider.Descriptor;
    }
    return _converter.ConvertTo(context, culture, value, destinationType);
}

public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType) {
    return sourceType == typeof(string) || _converter.CanConvertFrom(context, sourceType);
}

public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value) {
    string stringValue = value as string;
    
    if (stringValue != null) {
        if (stringValue.StartsWith("~/")) {                    
            // Assume this is a virtual path and return the instance description provider
            // for it
            return new VirtualPathInstanceDescriptorProvider(stringValue);
        }                
    }
    return _converter.ConvertFrom(context, culture, value);
}

First the parser asks the converter if it can convert from a string so we always say yes. In ConvertFrom we try to convert the value to a string and check if the path starts with “~/” in order to determine if it’s a virtual path. If it is a virtual path we return an object that knows how to get an InstanceDescriptor from the virtual path (VirtualPathInstanceDescriptorProvider). Now we have successfully parsed the control.

Next the code generator tries to generate code for the ITemplate property. The code generator will eventually ask if it can convert the object we returned earlier (VirtualPathInstanceDescriptorProvider) to an InstanceDescriptor. The implementation of VirtualPathInstanceDescriptorProvider returns an instance descriptor that wraps a constructor info for a custom template we are going to use:

internal class VirtualPathInstanceDescriptorProvider : IInstanceDescriptorProvider {
    private string _virtualPath;
    private static ConstructorInfo s_Constructor = GetConstructor();

    private static ConstructorInfo GetConstructor() {
        return typeof(VirtualPathTemplate).GetConstructor(new[] { typeof(string) });
    }

    public InstanceDescriptor Descriptor {
        get {
            return new InstanceDescriptor(s_Constructor, new[] { _virtualPath });
        }
    }

    public VirtualPathInstanceDescriptorProvider(string virtualPath) {            
        _virtualPath = virtualPath;
    }        
}

The code generation engine then generates the resulting code we wanted to specify above using the constructor info and virtual path.

At runtime we use BuildManager.CreateInstanceFromVirtualPath(“~/MyUserControl.ascx”) to create an instance of the user control and add it to the control’s collection and we’re done.

public class VirtualPathTemplate : ITemplate {
    private string _virtualPath;
    public VirtualPathTemplate(string virtualPath) {
        _virtualPath = virtualPath;
    }

    public void InstantiateIn(Control container) {
        // Try to create the control from the virtual path
        Control control = (Control)BuildManager.CreateInstanceFromVirtualPath(_virtualPath, typeof(Control));

        if (control != null) {
            // Add it to the controls collection
            container.Controls.Add(control);
        }            
    }
}

Caveats

In order to get the parser to use your TypeDescriptionProvider you have to make one call (which is nicely wrapped up in an API in the sample ParserConfiguration.RegisterTemplateOverride()) which registers a TypeDescriptionProvider for the control type so that it will work with any control that has an ITemplate property. If you are writing custom controls you can also specify the TypeConverter attribute and use this same trick to support this without the global override.

Also design time support is busted with this, but hey its a hack what would you expect :).

Check out the sample here.

Note: Code sample was built using vs2010 beta 2 but nothing should stop it from working on 3.5 sp1.

More Data Binding

davidfowl — Sun, 15 Nov 2009 07:54:56 GMT

Last time I tried to solve one of the deficiencies of data binding by taking advantage of expando attributes. Today I want to throw an idea out there that I’ve been playing with since that blog post.I was looking at WPF’s data binding and wondered what it would take to have data binding in ASP.NET be as first class as data binding in WPF.

There is no imperative (unless you mimic the generated code) way to setup 2 way data binding in ASP.NET, but what if there was? Maybe we should be able to setup bindings in the markup as well as code.

SetBinding

This hypothetical re-design of ASP.NET data binding includes a SetBinding method on the base control class that allows users to specify bindings imperatively.

public partial class _Default : System.Web.UI.Page {
    protected void Page_Init() {            
        textBox.SetBinding("Text", new DataBinding("FirstName"));
    }
}

The above code would setup 2 way binding for the text property to the FirstName property in the current item being databound (if we were using some data control).

<asp:TextBox runat="server" Text="{Binding FirstName}"></asp:TextBox>

This is the equivalent markup of the code above.

Unlike the current data binding story that’s all parse time magic, all the work here would be done in the runtime. The binding syntax would just be syntactic sugar.

So we’ve re-implemented the current binding that ASP.NET supports today but what else can we do. Let’s take this concept of a binding and extend it.

Sometimes it’s useful to have control’s properties be dependant on each other. Imagine some UI where you have a pager and a drop down list of possible page sizes. Whenever the page size changes you want to update the pager’s page size so that the list updates and shows the correct number of items. This can be done today with a bunch of code but wouldn’t it be nice to have this support natively in the framework?

<asp:ListView ID="ProductsList" runat="server">
    <ItemTemplate> 
        ...               
    </ItemTemplate>
</asp:ListView>
 
<asp:DataPager runat="server" 
    PagedControlID="ProductsList" 
    PageSize="{ControlBinding pageSizes, Property=SelectedValue}">
</asp:DataPager>        
 
<asp:DropDownList ID="pageSizes" runat="server">
    <asp:ListItem Text="10"></asp:ListItem>
    <asp:ListItem Text="20"></asp:ListItem>
    <asp:ListItem Text="30"></asp:ListItem>
    <asp:ListItem Text="40"></asp:ListItem>
</asp:DropDownList>

We’re using a control binding to bind the pageSizes.SelectedValue property to the pager’s PageSize property.

The prototype

Can’t have a blog post without code right :)? I’ve put together a little prototype of what this could look like but there are some gotchas.

There is only one bindable control in the prototype <asp:BindableTextBox> since I don’t have the power (I do but I wanted to give out a sample) to change the base Control class.
The binding syntax only works for text properties. The ASP parser doesn’t like it when you supply invalid values for properties. i.e <asp:BindableTextBox TextMode=”{Binding Foo}” /> won’t work since it will complain that it can’t convert that text to a TextMode enum.

However you have full power in the code behind to do all of these things.

To use the bindable controls just add the following to the <pages> section in web.config

<add tagPrefix="asp" namespace="Web.Binding.Controls" assembly="Web.Binding"/>

What’s in the package?

There is a BindableControl base class just in case you want to write more bindable controls :).

The BindableTextBox wraps a BindableControl since C# doesn’t support multiple inheritance and we want to get the behavior of both of those classes. Ideally this would be baked into the base Control class so all of the controls get this behavior for free.

The BindableControlBuilder parses the binding expressions and generates the right binding code.

public class BindableControlBuilder : ControlBuilder {
    private HashSet<BindingExpression> _bindings = new HashSet<BindingExpression>();
 
    public override void Init(TemplateParser parser, ControlBuilder parentBuilder, Type type, string tagName, string id, IDictionary attribs) {
        foreach (DictionaryEntry entry in attribs.Cast<DictionaryEntry>().ToList()) {
            string key = (string)entry.Key;
            string value = entry.Value as string;
            BindingExpression expr;
            if (value != null && BindingExpression.TryParse(key, value, out expr)) {
                // Add to our list of bindings
                _bindings.Add(expr);
                // Remove the attribute so the binding expression doesn't show up as a property value
                attribs.Remove(key);
            }
        }
        base.Init(parser, parentBuilder, type, tagName, id, attribs);
    }
 
    public override void ProcessGeneratedCode(CodeCompileUnit codeCompileUnit, CodeTypeDeclaration baseType, CodeTypeDeclaration derivedType, CodeMemberMethod buildMethod, CodeMemberMethod dataBindingMethod) {
        if (buildMethod != null) {
            foreach (var binding in _bindings) {
                // Generate code foreach binding and add it to the 
                // build method for this control
                var statement = binding.GenerateCode(buildMethod);
                int len = buildMethod.Statements.Count;
                buildMethod.Statements.Insert(len - 1, statement);
            }                
        }
    }
}

Also included in the package are 3 types of bindings:

Databinding – What ASP.NET has today.
ControlBinding – Bind a control property to another control’s property
ValueBinding – Bind a control property by executing a delegate to get the value.

The first 2 can be used declaratively but the ValueBinding is only supported imperatively.

I’d love to get feedback on what people think about this. You can download the prototype here.

Databinding 3.0

davidfowl — Fri, 13 Nov 2009 10:37:21 GMT

There was a post on our internal discussion group recently where a customer pointed out one of the weaknesses of 2 way data binding not working within user controls. Consider the following page:

<asp:ObjectDataSource runat="server" ID="personSource" 
    SelectMethod="GetPersons" 
    UpdateMethod="Update" 
    DataObjectTypeName="ExtendedDatabinding.Person"             
    TypeName="ExtendedDatabinding.PersonRepository">            
</asp:ObjectDataSource>
<asp:FormView runat="server" ID="personFormView" DataSourceID="personSource" DefaultMode="Edit" DataKeyNames="Id">
    <EditItemTemplate>
        <custom:personedit runat="server" ID="personForm" /> <br /> <br />                
        <asp:Button runat="server" Text="Update" CommandName="Update" />
    </EditItemTemplate>
</asp:FormView>

I have an ObjectDataSource that points to some business object and is bound to a form view. Also note that we have one usercontrol is the form view, <custom:personedit> which contains UI for an edit form. This is pretty useful since I may want the same UI for edit and insert I could just use the same usercontrol.

The problem is this doesn’t work with 2 way binding, so the FormView won’t extract the values from within the usercontrol even if there are <%# Bind() #> expressions declared. Here’s what the user control looks like:

<strong>First Name:</strong><asp:TextBox runat="server" ID="TextBox1" Text='<%# Bind("FirstName") %>'></asp:TextBox> <br /> <br />
<strong>Last Name:</strong><asp:TextBox runat="server" ID="TextBox2" Text='<%# Bind("LastName") %>'></asp:TextBox> <br /> <br />
<strong>Address:</strong><asp:TextBox runat="server" ID="TextBox3" TextMode="MultiLine" Rows="5" Text='<%# Bind("Address") %>'></asp:TextBox> <br /> <br />
<strong>State:</strong><asp:DropDownList runat="server" ID="DropDownList1" SelectedValue='<%# Bind("State") %>'> 
    <asp:ListItem Text=""></asp:ListItem>
    <asp:ListItem Text="FL"></asp:ListItem>
    <asp:ListItem Text="WA"></asp:ListItem>
    <asp:ListItem Text="CA"></asp:ListItem>
</asp:DropDownList>

The sad thing is all of those Bind expression will be ignored when we hit update.

I don’t want to get into exactly why this is the case (you should read about how bind works), instead I’d like to introduce a new way to do 2 way databinding.

In Dynamic Data 3.5 sp1 we introduced an interface IBindableControl which has an ExtractValues method that takes a dictionary to populate with name value pairs. FormView and ListView know about this interface and will look recursively for controls that implement IBindinableControl and call ExtractValues when performing an update, insert or delete (also to store the edit values in viewstate). We could make our usercontrol implement this interface and manually extract data from each control and put it into the dictionary, but that is tedious.

Enter databinding 3.0. Using the hidden gem ProcessGeneratedCode we can build a base class which I call BindableUserControl that supports a different Bind syntax to make this scenario work.

The idea exploits the fact that most controls derive from WebControl so we take advantage of their ability to use expando attributes and use it to declare a bogus Binding attribute with a some bind expression as the value. But enough talk lets dive into some code!

<%@ Control Language="C#" Inherits="Web.Binding.BindableUserControl" %>
 
<strong>First Name:</strong><asp:TextBox runat="server" ID="firstName" Binding="{Text=FirstName}"></asp:TextBox> <br /> <br />
<strong>Last Name:</strong><asp:TextBox runat="server" ID="lastName" Binding="{Text=LastName}"></asp:TextBox> <br /> <br />
<strong>Address:</strong><asp:TextBox runat="server" ID="address" TextMode="MultiLine" Rows="5" Binding="{Text=Address}"></asp:TextBox> <br /> <br />
<strong>State:</strong><asp:DropDownList runat="server" ID="ddl" Binding="{SelectedValue=State}"> 
    <asp:ListItem Text="FL"></asp:ListItem>
    <asp:ListItem Text="WA"></asp:ListItem>
    <asp:ListItem Text="CA"></asp:ListItem>
</asp:DropDownList>

We’re going to use the Binding attribute to determine what to bind and also the kind of binding to do. From just looking you can pretty much follow the syntax:
{ControlPropertyName=PropertyName}

There is also an enum you can use to specify what kind of binding you want to do:

{Text=Description, Mode=TwoWay}
{Text=Description, Mode=In}
{Text=Description, Mode=Out}

It basically tells the ControlBuilder what to generate i.e. Eval or ExtractValues statements or both.

The special base class BindableUserControl has a FileLevelControlBuilder that does all of the magic. The good news is that binding will work as expected now.

You can download the sample project here:

ExtendedDatabinding.zip

My first channel 9 video!

davidfowl — Fri, 23 Oct 2009 04:16:00 GMT

Wow, I haven't blogged in a while...but that will end soon. Watch me talk about the QueryExtender control. It's a new control we added in ASP.NET 4.0 to make Linq queries even simpler. The new control works with LinqDataSource and EntityDataSource. We also built all of the new DynamicData filters on top of this new control.

Check it out here

Dynamic Data in Regular Websites/Web Applications

davidfowl — Thu, 07 May 2009 01:33:00 GMT

Update: David Ebbo has a great video on channel9 about this. You can watch it here.

Today I'm excited to share that we've released DynamicData Preview 4 on codeplex. Check out the latest bits here.

This release is particularly interesting not only for people that have been using Dynamic Data for a while now, but anyone that has an existing application today who wants to use some of the niceties Dynamic Data offers without having to take all the junk associated. Take a look at the SimpleDynamicData project for examples.

Existing Sites

Here are 2 good reasons to use Dynamic Data:

Rich model validation
Rich templating support via FieldTemplates

If you've ever written a data driven app in webforms using our data controls, you would have realized that we are lacking a lot when it comes to validation. You can enable all of this goodness with a magic extension method.

protected void Page_Init() {

    GridView1.EnableDynamicData(your type here);

Note: The method requires that you pass in the type that may have annotations in order for us to read Metadata. If you're not familiar with the way annotations work in Dynamic Data then watch the videos here.

This method call enables DynamicControl/DynamicField to work within any of the data controls which makes use of FieldTemplates, and enables the rich validation support.

Making it work

So we know about this magic method call and somehow calling it with a type makes it all just work. Let's walk through an example of how we would use this. Here is my model:

public class Student {

    [Required]

    [Range(0, 100)]

    public int Age { get; set; }

    [Range(0.0, 4.0)]

    public double GPA { get; set; }

    [Required]

    public string FirstName { get; set; }

    [Required]

    public string LastName { get; set; }

    [DisplayFormat(DataFormatString = "{0:d/M/yyyy}")]

    public DateTime BirthDate { get; set; }

This is the type we are going to use for metadata. Using the attributes from System.ComponentModel.DataAnnotations we can add useful annotations to our model that will be used for validation and display formatting. Adding these attributes allows Dynamic Data to enable the appropriate validator. i.e RangeValidator, RequiredFieldValidator etc.

Now we're going to enable this on our GridView using the same method call as above in conjunction with an ObjectDataSource to complete our application:

protected void Page_Init() {

    GridView1.EnableDynamicData(typeof(Student));

Markup

<asp:GridView ID="GridView1" runat="server" DataSourceID="ObjectDataSource1" AutoGenerateEditButton="true">

    </asp:GridView>

    <asp:ObjectDataSource ID="ObjectDataSource1" runat="server"

        DataObjectTypeName="Student" DeleteMethod="DeleteStudent"

        InsertMethod="InsertStudent" SelectMethod="GetStudents"

        TypeName="StudentsRepository" UpdateMethod="UpdateStudent">

    </asp:ObjectDataSource>

Note: Dynamic Data takes care of the Metadata not the data. You still need databind the data control against some data source/data source control.

Here are the results when we are editing:

As you can see, the attributes we specified on our Student class directly affect the grid and validation is enabled.

There's more cool stuff to talk about but I'll mention those in upcoming posts. For now, download the preview and read up on Dynamic Data!