Dynamic LINQ Part 2 (Evolution)

A while ago, Scott Gu blogged about a Dynamic LINQ library that was part of the C# samples in VS2008. This library let developers put together late bound LINQ queries using a special syntax. This really comes in handy when you want to build a LINQ query based on dynamic input. Consider the following example:

public ActionResult List(string sort) {
    var northwind = new NorthwindEntities();
    IQueryable<Product> products = northwind.Products;
    switch (sort) {
        case "ProductName":
            products = products.OrderBy(p => p.ProductName);
            break;
        case "UnitsInStock":
            products = products.OrderBy(p => p.UnitsInStock);
            break;
        default:
            break;
    }

    return View(products.ToList());
}

In the above example, we’re trying to sort a list of products based on the sort parameter before we render the view. Writing the above code is painful, especially if you have a large number of sortable fields. Of course this isn’t the only way to write this code, in fact, if you’re a LINQ expert you can build an expression tree at runtime that represents the sort expression, but that solution is far from trivial to implement.


Last time we showed how we can (ab)use C# 4 dynamic feature to enable LINQ queries against dynamic types. Today I’ll show you how we can expand on that concept, and turn our previous solution into a general purpose library that allows us to execute late bound queries against existing LINQ providers.

The library itself is pretty small, it consists of 2 classes:

  • DynamicExpressionBuilder – The dynamic object that we used last time to build the actual expression tree
  • DynamicQueryable – a static class with a bunch of extension methods that are parallel to the actual Queryable static class

DynamicExpressionBuilder uses the same trick as last time (dynamic dispatch to build expressions), but this time we’re building expressions that would be recognized by most LINQ providers.

public override bool TryGetMember(GetMemberBinder binder, out object result) {
    result = new DynamicExpressionBuilder(Expression.Property(Expression, binder.Name));
    return true;
}

public override bool TryBinaryOperation(BinaryOperationBinder binder, object arg, out object result) {
    result = MakeBinaryBuilder(Expression, binder.Operation, arg);
    return true;
}

public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {           
    MethodInfo methodInfo = null;           
    if (args.Any(a => a == null)) {
        methodInfo = Expression.Type.GetMethod(binder.Name);
    }
    else {
        var types = args.Select(arg => arg.GetType()).ToArray();
        methodInfo = Expression.Type.GetMethod(binder.Name, types);
    }           

    if (methodInfo != null) {
        var expression = Expression.Call(Expression, methodInfo, args.Select(Expression.Constant));
        result = new DynamicExpressionBuilder(expression);
        return true;
    }
    return base.TryInvokeMember(binder, args, out result);
}
The actual implementation overrides more methods, but for brevity we’re only going to look at 3 of them. Each method captures the intent of the code 
and turns it into the appropriate
expression tree.
public static class DynamicQueryable {            
    public static IQueryable Where(this IQueryable source, Func<dynamic, dynamic> predicate) {
        return Where(source, GetExpression(source.ElementType, predicate));
    }

    private static LambdaExpression GetExpression(Type elementType, Func<dynamic, dynamic> expressionBuilder) {
        ParameterExpression parameterExpression = Expression.Parameter(elementType, expressionBuilder.Method.GetParameters()[0].Name);
        DynamicExpressionBuilder dynamicExpression = expressionBuilder(new DynamicExpressionBuilder(parameterExpression));
        Expression body = dynamicExpression.Expression;
        return Expression.Lambda(body, parameterExpression);
    }

    private static IQueryable Where(IQueryable source, LambdaExpression expression) {
        return source.Provider.CreateQuery(Expression.Call(typeof(Queryable),
                                                            "Where",
                                                            new[] { source.ElementType },
                                                            source.Expression,
                                                            Expression.Quote(expression)));
    }
    …etc
}
Normally when you’re writing LINQ queries, the implementations of Where, Select, SelectMany and other extension methods come from either the Enumerable or Queryable 
class.The DynamicQueryable class uses this pattern to provide overloads of the same extension methods on the non generic IQueryable.
This lets us do a lot of cool things with existing LINQ providers. Let’s look at some examples:
using System;
using System.Linq;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
Above we have a regular LINQ to Entities query against Northwind. Notice the usings are System, and System.Linq, those make sure that you’re using the Queryable 
implementations of Where and OrderBy.
We can write this same code in with Dynamic LINQ:
using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();
        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p.ProductName);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
We replaced the System.Linq using with a DynamicLINQ using, which makes use of our extensions methods. If we wanted to sort by a dynamic variable you could do it 
using the indexer syntax supported by DynamicQueryBuilder:
using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => p.UnitPrice < 10).OrderBy(p => p["ProductName"]);

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }
}
That’s all pretty cool, but there are more scenarios that can’t be done today using regular LINQ which Dynamic LINQ makes easy. 
One of the biggest annoyances for me using LINQ today, is the fact that custom functions can’t be used inside of query expressions. This makes total sense once 
you understand how LINQ expressions work, but if it were possible, you could make big queries a lot be more maintainable.
Take the following query using LINQ to Entities:
using System;
using System.Linq;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => IsCheap(p));

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }

    public static bool IsCheap(Product product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}
Anyone who has done any programming with LINQ to Entities knows that this won’t work. When we try to run this code, we get an error saying that this expression 
cannot be converted to SQL, or more precisely:

Unhandled Exception: System.NotSupportedException: LINQ to Entities does not recognize the method 'Boolean IsCheap(Sample.Product)' method, and this method cannot be translated into a store expression

The reason this doesn’t work is because the compiler puts the IsCheap method call in the expression tree and EF tries to convert it into SQL:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => IsCheap(p))

What we really wanted is for the compiler to replace IsCheap with the expression in the body of IsCheap, i.e. replace IsCheap with !product.Discontinued && product.UnitsInStock < 10.

This same expression would work today with DynamicLINQ:

using System;
using DynamicLINQ;

class Program {
    static void Main(string[] args) {
        NorthwindEntities northwind = new NorthwindEntities();

        var query = northwind.Products.Where(p => IsCheap(p));

        foreach (Product p in query) {
            Console.WriteLine(p.ProductName);
        }
    }

    public static dynamic IsCheap(dynamic product) {
        return !product.Discontinued && product.UnitsInStock < 10;
    }
}

We tweaked the method slightly, changing the argument and return type to be dynamic. When we run this code it works and we see a list of cheap products.

The difference here is that we’re actually executing code to build the expression tree, so instead of putting the method call into the resultant tree we actually invoke the method and the expression tree ends up looking like this:

Convert(value(System.Data.Objects.ObjectSet`1[Sample.Product])).MergeAs(AppendOnly).Where(p => (Not(p.Discontinued) And (Convert(p.UnitsInStock) < 10)))

There are also some scenarios that don’t work in the current implementation. For example, casting and some projections don’t work well with existing LINQ providers. I’ve put the code on a bitbucket repository.

http://bitbucket.org/dfowler/dynamiclinq

Enjoy!

Published Thursday, August 19, 2010 1:34 AM by davidfowl
Filed under: , ,

Comments

# re: Dynamic LINQ Part 2 (Evolution)

Thursday, August 19, 2010 2:21 PM by apocolpse

Very cool.

Take a look at this -> www.albahari.com/.../linqkit.aspx

LinqKit allows the use custom functions and expressions composition

# re: Dynamic LINQ Part 2 (Evolution)

Friday, August 20, 2010 12:28 PM by Fahad

Your example only uses extension methods that generate IL code, From what I have used, You should probably be following the pattern of IQueryable/ IQueryProvider to implement a full blown Dynamic LINQ, Or One more way is wrap around your methods with Expression<Func> instead of Func<>, that would be more appropriate.

-Fahad

# re: Dynamic LINQ Part 2 (Evolution)

Saturday, August 21, 2010 12:32 PM by davidfowl

@Fahad - That's actually a lot harder than it sounds. Also, you can't even have Expression<Func<,>> with dynamic in the mix, read my previous article.

# re: Dynamic LINQ Part 2 (Evolution)

Tuesday, August 31, 2010 11:08 PM by Ajai Shankar

This is beautiful!

I wish the people who were so quick to trash the earlier post would give this the attention it deserves.

Thanks

Ajai

# re: Dynamic LINQ Part 2 (Evolution)

Wednesday, October 20, 2010 5:32 AM by mbarbac

Very interesting.

You may want also to take a look at www.codeproject.com/.../KyneticORM.aspx, an ORM alternative without the need of any configuration.

Rgds, M

# re: Dynamic LINQ Part 2 (Evolution)

Monday, April 18, 2011 12:25 PM by MvcCmsJon

Hey David,

I added some ThenBy support and a few other things to the code at...

mvccms.codeplex.com/.../9335cc901cde

I was wanting to find a way to filter/search with this also with columns coming in from jquery.datatables.  I was able to get the type and dynamically add the expressions but I could not find a way to "or" join the expressions.  Any ideas on this?

# re: Dynamic LINQ Part 2 (Evolution)

Tuesday, April 19, 2011 3:45 AM by MvcCmsJon

I also added the ability for this to work with a predicate builder which allows from some pretty cool dynamic generic grid filtering.

I added this method

       private static Expression<Func<T, bool>> GetFuncTbool<T>(IQueryable source, Func<dynamic, dynamic> expressionBuilder)

       {

           ParameterExpression parameterExpression = Expression.Parameter(GetElementType(source), expressionBuilder.Method.GetParameters()[0].Name);

           DynamicExpressionBuilder dynamicExpression = expressionBuilder(new DynamicExpressionBuilder(parameterExpression));

           Expression body = dynamicExpression.Expression;

           return Expression.Lambda<Func<T, bool>>(body, parameterExpression);

       }

The full info on this is here.

www.ranessoftwareconsulting.com/.../reusable-dynamiclinq-with-predicatebuilder-to-search-every-column-of-jquerydatatables

# re: Dynamic LINQ Part 2 (Evolution)

Sunday, April 24, 2011 1:53 AM by davidfowl

@MvcCmsJon Nice! I'll look at what you have and try to build something based on it! That's awesome!

# re: Dynamic LINQ Part 2 (Evolution)

Friday, July 1, 2011 2:38 PM by LauraH

Could I use this library to join 2 IEnumerable<dynamic> objects?

I have the keys and the select columns in string form.

# re: Dynamic LINQ Part 2 (Evolution)

Saturday, April 13, 2013 5:04 AM by Courtney

Hey there just wanted to give you a quick heads up. The words in your article seem

to be running off the screen in Firefox. I'm not sure if this is a formatting issue or something to do with browser compatibility but I figured I'd post to let you know.

The design and style look great though! Hope you get the issue solved soon.

Thanks