Microsoft Israel's Developers' Academy #3

Voting has been open for a few days now for which lectures will be featured at the next Microsoft Developers' Academy. The event is the second most important event for Microsoft-platform-developers in Israel, surpassed only by the biennial Tech-Ed Israel.

Among some very interesting lectures I would love to see, I've suggested a lecture myself. Here's its abstract:

A Deep Dive into LINQ

Many developers already use LINQ on a daily basis, but most only scratch the surface of what’s possible. In this session we will dive more deeply into LINQ and see how it works behind the scenes, share tips, tricks and common pitfalls.

While most of the lecture's structure is already defined in my head, I'd love to hear from you what topics you would like to see covered, either from the topics covered in my blog's posts or from your own experience.

Go vote and I'll be seeing you there! :)

Visual Studio 2008 Load Testing Checklist

After a couple of days of trying to run a load test for a web service on several agents via Visual Studio 2008, I come out much wiser and with a few new bald-spots, where hair I pulled out in the process used to be.

I got a few errors whose messages have nothing to do with what really happened, so here's a quick checklist:

• Try restarting the agent service.
• Re-register the agent using AgentConfigUtil and then restart the agent service.
• Try restarting the controller service.
• Are you using a trial version of Visual Studio Team System? If so, it may have expired.
• Are all of your agents and controller on the same version?
  Check that the file Microsoft.VisualStudio.QualityTools.LoadTestFramework.dll is the same version on all of your computers (more information). Its original version is 9.0.21022.8, SP1's version is 9.0.30729.1. If you installed the trial from Microsoft's site, your version is pre-SP1.
  If any of the agents differs from the controller, you will not be able to use them and will see them as Disconnected in the Administer Test Controllers dialog.
• Are you using SQL Server as a store for the results?
• Does the user the service runs as have permissions to the SQL Server database?
  You can see which user is used to access the store from Visual Studio's Menu Test -> Administer Test Controllers.
• If you're getting login errors, check that the SQL Server allows remote connections. If it does, consider using a SQL Server user instead of Windows Authentication. Remember to make sure the SQL Server allows both types of connections (Mixed Mode) before attempting this.
• Make sure the computer from which the SQL Server is running has an exception for incoming connections on TCP port 1443.
• Using the trial version of the agents? Getting "The Visual Studio Team Test Load Agent license has expired" way before your 90 day trial period has elapsed? You've probably hit the 25 tests mark (note that the tests are per-CPU, rather than per-computer).
  You can work around this if you change the user running the service.

Around that time I decided I've had enough with all this trial stuff. I was thinking about purchasing licenses, but then I took a look and found that you can only use load agents when you've got a Volume License!

At that point, I went and just ran the test on a few computers at once.

Extending The Partial Classes of Stored Procedures' Results in LINQ to SQL

One of the nice things about LINQ to SQL is the ability to extend the types of the generated entities. Another nice thing is being able to get typed results from stored procedures. Let's try and combine the two together.

We'll take the following generated code for instance:

public partial class GetStuffResult
{
    
    private int _Id;
    
    private string _Name;
    
    public GetStuffResult()
    {
    }
    
    [Column(Storage="_Id", DbType="INT NOT NULL")]
    public int Id
    {
        get
        {
            return this._Id;
        }
        set
        {
            if ((this._Id != value))
            {
                this._Id = value;
            }
        }
    }
    
    [Column(Storage="_Name", DbType="NVarChar(100)")]
    public string Name
    {
        get
        {
            return this._Name;
        }
        set
        {
            if ((this._Name != value))
            {
                this._Name = value;
            }
        }
    }
}

I would like to extend it by adding a new property to it, named TranslatedId, which translates the returned Id to and from a Guid:

partial class GetStuffResult
{
    public Guid TranslatedId
    {
        get
        {
            return Translator.GetGuid(this.Id);
        }
        set
        {
            this.ProjectId = Translator.GetInt(value);
        }
    }
}

Translator's two methods translate between the integer id and the Guid id. Note that 0 translates to and from an empty Guid.

I'll try to run it and will find out that all of my Ids are 0. Why is that?

Apparently, when running a stored procedure in LINQ to SQL, it requires all of the properties defined on the class to have both getters and setters (which means you can not write read-only properties in the partial class) and sets all non-result (i.e. non-generated) properties to their default values. This means that although the real Id gets selected from the database, TranslatedId gets set to an empty Guid immediately afterwards, which in turn overrides Id to 0.

To work around this, we have to apply a dirty little hack:

partial class GetStuffResult
{
    private Guid dupe;

    [Column(Storage = "dupe")]
    public Guid TranslatedId
    {
        get
        {
            return Translator.GetGuid(this.Id);
        }
        set
        {
            this.ProjectId = Translator.GetInt(value);
        }
    }
}

This is pretty horrible, but what it does is direct LINQ to SQL to set the default value into the property's storage field, which is some dupe field, instead of into the property itself. Now the field's value will be reset and not the property's. Note that the dupe field's type should be the same as the property's type.

Yes, it's ugly, but this is the only workaround I found. If anyone knows of any other workaround or solution to this problem, I'd love to hear it.

[Original image used: Where do we go next?]

Announcing New C# 4.0 LINQ Features and Book

* Fabrice Marguerie, co-author of LINQ In Action, asked me to clarify that this post has absolutely nothing to do with their book and is simply meant in jest *

LINQ has been around for quite a while, making our lives easier with its short, declarative syntax. During talks in closed sessions, the persons behind C# have come up with some great plans for its next version, including some vast improvements to LINQ. The team wanted to make everyone's lives even better by introducing some ground-breaking features to it.

I'd like to introduce, out of those finalized or at the final draft stage, some of the new keywords that have been introduced to the language with regards to LINQ and by doing so, expose the reader to this new, exciting technology.

fine

Many of the lines of code incorporated into LINQ are about variable validation. How many times have you had to write pieces of code like this:

var values = from n in numbers
             where n > 0 && n < 10
             select n;

This piece of code is very trivial and the compiler is smart enough to understand it on its own. From C# 4.0, the fine keyword is introduced and the above query will be replaced with:

var values = from n in numbers
             where n is fine
             select n;

some

Most LINQ queries do not iterate over an entire enumeration. Some filter out results, some use Take, Skip and other methods in order to run on only a subset of the elements. Since this is an extremely common scenario, the team has decided to integrate it into the language using the new some keyword:

var values = from n in some numbers
             select n;

This is complemented with the new most keyword, which is like some, only it takes more elements.

somewhere

The from clause is known to all who use LINQ, as it indicates the source of the data. Since this clause is a recurring pattern (statistical surveys have shown it to occur 99% of the time, allowing for a 4% standard deviation), the team behind the new language features has decided to cut it out, thus removing unnecessary verbosity from the language. This, in turn, will cause less characters to be written and rewritten, meaning less keystrokes, less keyboard wear, less purchases of new keyboards, leading to prevention of global warming, a goal near to Microsoft's heart. You can see Microsoft's predictions for this in the graph to the right.

This is possible due to the new somewhere keyword, as described below:

var values = from somewhere
             select n;

This feature relies heavily upon the newly introduced Type Inference Transfer System (abbreviated MANTA), a technology recently demonstrated by Microsoft Research.

The Book

I've long since wanted to write a book about something. Since this is just the tip of the iceberg, I've decided to write my book about this subject. It will contain most of what I know (hey, I still need to speak about something at conferences and release a second edition some day, right?) about the new language. I think that with all of the new features, making our lives as developers that much easier, I've picked an appropriate title.

Here's an excerpt from the introduction:

Recent advances in compilation technologies have brought with them less need for verbosity. In its stead come smart compilers, which infer much of the work for the developer. With such technology, there's not a lot you, as a developer, need to do anymore. You can simply write a short statement, validate its correctness, check the code in and go home for the day.

Welcome to the world of tomorrow. Put your feet up.

Here's the layout of the book, as I plan it:

1. Introduction
2. New Features in LINQ

What else would you like to see in the book? Let me know and I might just send you a copy of the rough draft for review once I finish it.

Throw Before You Yield

In a comment left by Bart De Smet, he pointed out that I failed to address the fact that the execution of all "yielding" methods is deferred.

For instance, when running the following code, no exceptions will be thrown:

int[] arr = null;
var copy = arr.Enumerate();



// ...

static IEnumerable<T> Enumerate<T>(this IEnumerable<T> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    foreach (var item in enumeration)
    {
        yield return item;
    }
}

This is because the first time the code from the method will run will be after there a call to copy's GetEnumerator(), followed by a call to that object's MoveNext() method has been made (i.e. when we enumerate over copy).

To overcome this problem, we'll change the code slightly:

static IEnumerable<T> Enumerate<T>(this IEnumerable<T> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    return EnumerateCore(enumeration);
}

private static IEnumerable<T> EnumerateCore<T>(IEnumerable<T> enumeration)
{
    foreach (var item in enumeration)
    {
        yield return item;
    }
}

Now a NullReferenceException will immediately be thrown out of Enumerate, since the "unyielding" method will first run the code and then defer-call to the "yielding" method. This helps our "yielding" methods adhere to the principal of fail-fast.

MSDN Wiki?

I just love the fact that there's a "Community Content" section to the MSDN documentation. I just found out that there's an undocumented exception coming from the System.Net.Mail.MailMessage.Subject property, I browsed to the relevant MSDN page and simply added a line saying so.

It's a nice copy of the features from PHP's Documentation, but about 10 years too late, and also, mind you, not a Wiki at all. It's comments. In a real Wiki, my corrections to the documentation would be displayed inline with the original documentation. You know, like Wikipedia. Dear Microsoft, please turn it into a real Wiki, at which time your documentation will retain its relevance for me.

On a side note, does anyone know of an add-in for Visual Studio that replaces F1 to a call to Google in my default browser?

Pitfall: Static Field Inline Initialization Order of Execution

public class StaticDemo
{
    static readonly int Sum = A + B;
    static readonly int A = Calculator.GetA();
    static readonly int B = Calculator.GetB();
}

This code will initialize A and B from the Calculator class and initialize Sum from the sum of A and B. However, when you run this code, Sum will equal 0, no matter what A or B are.

Why is this? Apparently, when initializing static fields inline, the order of execution is from top to bottom, regardless of dependencies. Trying to use the same method to initialize an instance field will result in the error A field initializer cannot reference the non-static field, method, or property.

So how do we solve this, you ask?

public class StaticDemo
{
    static readonly int A = Calculator.GetA();
    static readonly int B = Calculator.GetB();
    static readonly int Sum = A + B;
}

Extension Methods Roundup: Intersect, Union, AsNullable and GroupEvery

Here we go with the third installment of the Extension Method Roundup. The reason behind these 'code dumps' is that LINQ is a central part of my coding and always find new problems I want to find elegant solutions to. Hope these prove as useful to you as they do to me.

Intersect / Union

Again, shorthand for when you have Enumerable of Enumerable of T and you simply want to intersect or union all of the enumerations in one single call.

public static IEnumerable<T> Intersect<T>(this IEnumerable<IEnumerable<T>> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    IEnumerable<T> returnValue = null;

    foreach (var e in enumeration)
    {
        if (returnValue != null)
            returnValue = e;
        else
            returnValue = returnValue.Intersect(e);
    }

    return returnValue;
}

public static IEnumerable<T> Union<T>(this IEnumerable<IEnumerable<T>> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    IEnumerable<T> returnValue = null;

    foreach (var e in enumeration)
    {
        if (returnValue != null)
            returnValue = e;
        else
            returnValue = returnValue.Union(e);
    }

    return returnValue;
}

AsNullable

I was always missing this method, to coincide with the Cast and OfType methods.

public static IEnumerable<T?> AsNullable<T>(this IEnumerable<T> enumeration)
    where T : struct
{
    return from item in enumeration
           select new Nullable<T>(item);
}

GroupEvery

This takes count items from an enumeration and groups them into a single array.

public static IEnumerable<T[]> GroupEvery<T>(this IEnumerable<T> enumeration, int count)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    if (count <= 0)
        throw new ArgumentOutOfRangeException("count");

    int current = 0;
    T[] array = new T[count];

    foreach (var item in enumeration)
    {
        array[current++] = item;

        if (current == count)
        {
            yield return array;
            current = 0;
            array = new T[count];
        }
    }

    if (current != 0)
    {
        yield return array;
    }
}

I've also gone and updated my LINQ Extensions project on CodePlex with everything I've published since the last update. You're welcome to download and fiddle with it. :)

Asynchronously Preloaded LINQ Queries

As a rule of thumb, when presented with two independent blocking I/O operations on more than one independent devices, it's best to use threads to create parallel operations, instead of waiting for a single synchronous operation to complete. That way, executing operations O1, ..., On, each of which take T1, ..., Tn will result in total time where T < T1 + ... + Tn, instead of T = T1 + ... + Tn.

Let's take a simple example that consists of the following two operations: The first tries to load all of the files in the My Pictures folder as assemblies (silly), while the other simulates some obscure database operation by calling Thread.Sleep (very silly).

var assemblies = (from file in new DirectoryInfo(@"C:\...\My Pictures").GetFiles("*.*")
                  let assembly = TryLoadAssembly(file.FullName)
                  where assembly != null
                  select assembly).ToArray();

// Do some long database work.
Thread.Sleep(1000);

The code executes synchronously in about two seconds, one second for assembly loading operation and another second holding the thread.

What we could do is enqueue a work item for the ToArray call before the database operation and complete the execution synchronously once we're done. I've coded a short-hand syntax for that:

var assemblies = from file in new DirectoryInfo(@"C:\...\My Pictures").GetFiles("*.*")
                 let assembly = TryLoadAssembly(file.FullName)
                 where assembly != null
                 select assembly;

assemblies = assemblies.Buffered();

// Do some long database work.
Thread.Sleep(1000);

// Force Load
assemblies = assemblies.ToArray();

Once the Buffer call is made, the deferred query begins to run in a separate thread, buffering items into memory, waiting for the query to be executed. Once the query is executed, the buffered items are immediately returned and the iteration completes synchronously. The above code takes approximately one second to run, as both operations run concurrently.

I've attached the code behind this for your consideration and will be adding it to my LINQ Extensions project on CodePlex once I get around to it... :)

public static class ExtensionMethods
{
    /// <summary>
    /// Asynchronously begins buffering an enumeration, even before it is lazy loaded.
    /// </summary>
    public static IEnumerable<T> Buffered<T>(this IEnumerable<T> enumeration)
    {
        // Check to see that enumeration is not null
        if (enumeration == null)
            throw new ArgumentNullException("enumeration");

        return new AsyncEnumerable<T>(enumeration);
    }

    private class AsyncEnumerable<T> : IEnumerable<T>
    {
        private volatile bool shouldContinueBuffering;
        private IEnumerator<T> enumerator;
        private IAsyncResult asyncResult;
        private List<T> buffer;
        private Action bufferAction;
        private object syncLock;

        public AsyncEnumerable(IEnumerable<T> enumeration)
        {
            this.enumerator = enumeration.GetEnumerator();
            this.shouldContinueBuffering = true;
            this.buffer = new List<T>();
            this.syncLock = new object();
            this.bufferAction = this.Buffer;

            this.asyncResult = this.bufferAction.BeginInvoke(null, null);
        }

        private void Buffer()
        {
            lock (this.syncLock)
            {
                // Continue buffering for as long as we can and while there are still items left.
                while (this.shouldContinueBuffering && this.enumerator.MoveNext())
                {
                    buffer.Add(enumerator.Current);
                }
            }
        }

        IEnumerator<T> IEnumerable<T>.GetEnumerator()
        {
            this.shouldContinueBuffering = false;

            // Wait for the last item buffered to finish.
            lock (this.syncLock)
            {
                // End invocation so that exceptions could be throw here.
                this.bufferAction.EndInvoke(this.asyncResult);
            }

            // Iterate over buffered items.
            foreach (var item in buffer)
            {
                yield return item;
            }

            // Continue iterating from the point where we stopped buffering.
            while (enumerator.MoveNext())
            {
                yield return enumerator.Current;
            }
        }

        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
        {
            return ((IEnumerable<T>)this).GetEnumerator();
        }
    }
}

Extension Methods Roundup: Remove, Aggregate, At, AsIndexed and Friends

Hey hey hey! It's time for another Extension Methods Roundup! Here are some of the extension methods I've written since the last one:

Dictionary's Missing Remove Methods

public static void Remove<TKey, TValue>(this IDictionary<TKey, TValue> dictionary, TValue value)
{
    // Check to see that dictionary is not null
    if (dictionary == null)
        throw new ArgumentNullException("dictionary");

    foreach (var key in (from pair in dictionary
                         where EqualityComparer<TValue>.Default.Equals(value, pair.Value)
                         select pair.Key).ToArray())
    {
        dictionary.Remove(key);
    }
}

public static void RemoveRange<TKey, TValue>(this IDictionary<TKey, TValue> dictionary, IEnumerable<TValue> values)
{
    // Check to see that dictionary is not null
    if (dictionary == null)
        throw new ArgumentNullException("dictionary");

    // Check to see that values is not null
    if (values == null)
        throw new ArgumentNullException("values");

    foreach (var value in values.ToArray())
    {
        ExtensionMethods.Remove(dictionary, value);
    }
}

public static void RemoveRange<TKey, TValue>(this IDictionary<TKey, TValue> dictionary, IEnumerable<TKey> keys)
{
    // Check to see that dictionary is not null
    if (dictionary == null)
        throw new ArgumentNullException("dictionary");

    // Check to see that keys is not null
    if (keys == null)
        throw new ArgumentNullException("keys");

    foreach (var key in keys.ToArray())
    {
        dictionary.Remove(key);
    }
}

String Aggregation

public static string Aggregate(this IEnumerable<string> enumeration, string separator)
{
    return Aggregate(enumeration, str => str, separator);
}

public static string Aggregate<T>(this IEnumerable<T> enumeration, Func<T, string> toString, string separator)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    // Check to see that toString is not null
    if (toString == null)
        throw new ArgumentNullException("toString");

    // Check to see that separator is not null or an empty string
    if (string.IsNullOrEmpty(separator))
        throw new ArgumentNullException("separator");

    return enumeration.Aggregate(string.Empty,
        (accum, item) => string.Format("{0}{1}{2}", accum, separator, toString(item)),
        str => str.Length > separator.Length ? str.Substring(separator.Length) : str);
}

Those are very good for when you want to create strings such as "a, b, c, d".

LastOrDefault

public static T LastOrDefault<T>(this IList<T> list)
{
    // Check to see that list is not null
    if (list == null)
        throw new ArgumentNullException("list");

    if (list.Count == 0)
        return default(T);

    return list[list.Count - 1];
}

This is an optimized version of the original LastOrDefault for lists that allow random access.

At

public static T At<T>(this IEnumerable<T> enumeration, int index)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    return enumeration.Skip(index).First();
}

public static IEnumerable<T> At<T>(this IEnumerable<T> enumeration, params int[] indices)
{
    return At(enumeration, (IEnumerable<int>)indices);
}

public static IEnumerable<T> At<T>(this IEnumerable<T> enumeration, IEnumerable<int> indices)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    // Check to see that indices is not null
    if (indices == null)
        throw new ArgumentNullException("indices");

    int currentIndex = 0;

    foreach (int index in indices.OrderBy(i => i))
    {
        while (currentIndex != index)
        {
            enumeration = enumeration.Skip(1);
            currentIndex++;
        }

        yield return enumeration.First();
    }
}

At provides pseudo-random access to enumerable lists, where needed. I've found use for it in a couple of places which returned indices for non-IList<T> enumerations.

SequenceEqual<T1, T2>

public static bool SequenceEqual<T1, T2>(this IEnumerable<T1> left, IEnumerable<T2> right, Func<T1, T2, bool> comparer)
{
    using (IEnumerator<T1> leftE = left.GetEnumerator())
    {
        using (IEnumerator<T2> rightE = right.GetEnumerator())
        {
            bool leftNext = leftE.MoveNext(), rightNext = rightE.MoveNext();

            while (leftNext && rightNext)
            {
                // If one of the items isn't the same...
                if (!comparer(leftE.Current, rightE.Current))
                    return false;

                leftNext = leftE.MoveNext();
                rightNext = rightE.MoveNext();
            }

            // If left or right is longer
            if (leftNext || rightNext)
                return false;
        }
    }

    return true;
}

This differs from the original SequenceEqual in that it is able to accept two different types of sequences.

AsIndexed

public static IEnumerable<KeyValuePair<int, T>> AsIndexed<T>(this IEnumerable<T> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    int i = 0;

    foreach (var item in enumeration)
    {
        yield return new KeyValuePair<int, T>(i++, item);
    }
}

This is when you need indices, but don't want the overhead of creating an Array<T>.

The Missing SelectMany

public static IEnumerable<T> SelectMany<T>(this IEnumerable<IEnumerable<T>> source)
{
    // Check to see that source is not null
    if (source == null)
        throw new ArgumentNullException("source");

    foreach (var enumeration in source)
    {
        foreach (var item in enumeration)
        {
            yield return item;
        }
    }
}

Oh, come on! Why wasn't there a parameterless SelectMany in the framework? Oh well, here's one.

ToDictionary of IGrouping

public static Dictionary<TKey, IEnumerable<TElement>> ToDictionary<TKey, TElement>(
    this IEnumerable<IGrouping<TKey, TElement>> enumeration)
{
    // Check to see that enumeration is not null
    if (enumeration == null)
        throw new ArgumentNullException("enumeration");

    return enumeration.ToDictionary(item => item.Key, item => item.Cast<TElement>());
}

This is shorthand for when you want to create a dictionary from the result of GroupBy.