Kenny Kerr : Introduction to MSIL

Introduction to MSIL – Part 8 – The for each Statement

KennyKerr — Wed, 15 Dec 2004 16:26:00 GMT

The for each statement; popularized by Visual Basic, humbly accepted by C++, and forever immortalized by a creation lovingly known as ECMA-334 (OK so some people simply call it C#).

In this installment of my open-ended series entitled Introduction to MSIL, I am talking about one of the most popular statements in commercially successful programming languages. Whether your focus has been on Visual Basic, COM, Standard C++, or .NET (or all of the above) you’ve inevitably come across some form of the for each statement.

You may have noticed a trend in this series. As we dig deeper into more interesting constructs, the focus is less and less on MSIL instructions and more on language design and compiler implementations. There is really nothing especially new in the instructions you might use to implement a for each statement. Nevertheless it is useful to understand what makes the statement work and how it is interpreted and converted to MSIL instructions depending on the context in which it is used. This is where you thank your lucky stars that your boss lets you write code in a language with statements that abstract away the dirty secrets hidden behind their elegant exteriors. for each is just such a statement.

Variations of the for each statement, as it relates to .NET, can be found in C#, Visual Basic .NET and C++/CLI. There may also be other languages with a similar statement that I’m not aware of. The examples in this part of the series use C++/CLI only because C# and VB get enough press as it is. As I’ve said before, I am not focusing on a particular compiler’s implementation, but rather demonstrating the techniques that may be used by compilers in general.

Let’s start with a simple example:

array<int>^ numbers = gcnew array<int> { 1, 2, 3 };

for each (int element in numbers)
{
Console::WriteLine(element);
}

numbers is a synthesized reference type extending the built-in System.Array type. It is initialized with three elements and the for each statement executes the statements in the scope block for each of the elements. Pretty straight forward; I’m sure you can imagine the equivalent code in your language of choice. Quite predictably this can be implemented in terms of the following set of instructions:

    .locals init (int32[] numbers,
                  int32 index)

// Create the array

    ldc.i4.3
    newarr int32
    stloc numbers

// Populate the array

    ldloc numbers
    ldc.i4.0 // index
    ldc.i4.1 // value
    stelem.i4

    ldloc numbers
    ldc.i4.1 // index
    ldc.i4.2 // value
    stelem.i4

    ldloc numbers
    ldc.i4.2 // index
    ldc.i4.3 // value
    stelem.i4

    br.s _CONDITION

_LOOP:

    ldc.i4.1
    ldloc index
    add
    stloc index

_CONDITION:

    ldloc numbers
    ldlen
    ldloc index
    beq _CONTINUE

// for each statements

    ldloc numbers
    ldloc index
    ldelem.i4
    call void [mscorlib]System.Console::WriteLine(int32)

    br.s _LOOP

_CONTINUE:

The only instructions not already discussed in this series are those related to arrays. The newarr instruction pops an integer off the stack indicating the number of elements in the array. It then creates the array of the given type and pushes a reference to it onto the stack. The stelem instruction replaces an element in an array with a given value. The stack needs to be prepopulated with a reference to the array, the index of the element in the array, and the value to store at the given position. Finally, the ldelem instruction loads an element from an array and pushes it onto the stack by first popping off a reference to an array as well as the index into the array.

As you can see, this is not unlike the code that might be generated for a for statement looping over the same elements, as discussed in part 6. Of course the for each statement can be used with more than just types deriving from System.Array. In fact it is capable of enumerating the elements of any type that either implements the IEnumerable interface or even a type providing the same functionality without actually implementing that interface. The degree of support naturally depends on the scope of your language specific implementation. Some languages could even, nay, do provide support for completely unrelated containers in the for each statement. Consider the following example using the Standard C++ Library vector container class:

std::vector<int> numbers;
numbers.reserve(3);
numbers.push_back(1);
numbers.push_back(2);
numbers.push_back(3);

for each (int element in numbers)
{
std::cout << element << std::endl;
}

In this example, the compiler calls the vector’s begin method to get an iterator pointing to the first element in the container. This iterator is then used to enumerate the elements, incrementing the iterator until it compares equal to the iterator returned by vector’s end method. For each iteration, the iterator is dereferenced with the resulting value assigned to the element variable. The beauty here is that using the for each statement avoids many common errors related to buffer overruns and the like.

Let’s get back to managed code. If the for each statement has to support collections other than arrays, surely this leads to different implementations depending on the use of the statement. Consider the following example:

Collections::ArrayList numbers(3);
numbers.Add(1);
numbers.Add(2);
numbers.Add(3);

for each (int element in %numbers)
{
Console::WriteLine(element);
}

As an aside, keep in mind that most of the Standard C++ Library containers that include a constructor taking a single integer typically set the initial size of the container, whereas the .NET collection types use this constructor to set the initial capacity.

So what does for each do in this case? Well the ArrayList type implements the IEnumerable interface with its single GetEnumerator method. The for each statement calls the GetEnumerator method to get an implementation of the IEnumerator interface that is then used to enumerate the collection. Let’s examine a simple implementation for this code:

    .locals init (class [mscorlib]System.Collections.ArrayList numbers,
                  class [mscorlib]System.Collections.IEnumerator enumerator)

// Create the array

    ldc.i4.3
    newobj instance void [mscorlib]System.Collections.ArrayList::.ctor(int32)
    stloc numbers

// Populate the array

    ldloc numbers
    ldc.i4.1
    box int32
    callvirt instance int32 [mscorlib]System.Collections.ArrayList::Add(object)
    pop

    ldloc numbers
    ldc.i4.2
    box int32
    callvirt instance int32 [mscorlib]System.Collections.ArrayList::Add(object)
    pop

    ldloc numbers
    ldc.i4.2
    box int32
    callvirt instance int32 [mscorlib]System.Collections.ArrayList::Add(object)
    pop

// Get the enumerator

    ldloc numbers

    callvirt instance class [mscorlib]System.Collections.IEnumerator
        [mscorlib]System.Collections.IEnumerable::GetEnumerator()

    stloc enumerator

    br.s _CONDITION

_CONDITION:

    ldloc enumerator
    callvirt instance bool [mscorlib]System.Collections.IEnumerator::MoveNext()
    brfalse.s _CONTINUE

// for each statements

    ldloc enumerator
    callvirt instance object [mscorlib]System.Collections.IEnumerator::get_Current()
    call void [mscorlib]System.Console::WriteLine(object)

    br.s _CONDITION

_CONTINUE:

Well that looks good, but there’s something missing. Can you tell what it is? Since the for each statement is resulting in a new object being created by virtue of calling the GetEnumerator method, it is conceivable that this object requires deterministic cleanup to avoid resource leaks. The enumerator expresses this by implementing the IDisposable interface. Unfortunately, the compiler is not always capable of discerning this at compile time, though if the compiler has enough static type information there is certainly value in taking advantage of it to optimize the generated instructions. As it turns out, the enumerator provided by the ArrayList type does not implement the IDisposable interface. Of course this could change in future so “optimizing” your code for this type of thing is not a wise move. Compilers can use the isinst instruction to determine whether the enumerator implements the IDisposable interface. Arguably this is an oversight in the design of the IEnumerable and IEnumerator interfaces.

To address this quirk, the generic IEnumerator interface introduced in version 2.0 of the .NET Framework makes it explicit by deriving from IDisposable so implementations have to provide a Dispose method even if it does nothing. This certainly simplifies matters for the caller. Collections.Generic.IEnumerator<T> is the return type for the GetEnumerator method from the Collections.Generic.IEnumerable<T> interface implemented by the new generic collection types. For compatibility with existing code, the enumerator implementations also implement the older, non-generic IEnumerator interface. Consider the following example:

Collections::Generic::List<int> numbers(3);
numbers.Add(1);
numbers.Add(2);
numbers.Add(3);

for each (int element in %numbers)
{
Console::WriteLine(element);
}

In this case there is no question of whether the enumerator implements IDisposable. The only issue remaining is coming up with a reliable way of calling the Dispose method in the face of exceptions. Here’s a reasonable solution using the exception handling constructs discussed in part 5.

    .locals init (class [mscorlib]System.Collections.Generic.'List`1'<int32> numbers,
                  class [mscorlib]System.Collections.Generic.'IEnumerator`1'<int32> enumerator)

// Create the array

    ldc.i4.3
    newobj instance void class [mscorlib]System.Collections.Generic.'List`1'<int32>::.ctor(int32)
    stloc numbers

// Populate the array

    ldloc numbers
    ldc.i4.1 // value
    callvirt instance void class [mscorlib]System.Collections.Generic.'List`1'<int32>::Add(!0)

    ldloc numbers
    ldc.i4.2 // value
    callvirt instance void class [mscorlib]System.Collections.Generic.'List`1'<int32>::Add(!0)

    ldloc numbers
    ldc.i4.3 // value
    callvirt instance void class [mscorlib]System.Collections.Generic.'List`1'<int32>::Add(!0)

// Get the enumerator

    ldloc numbers

    callvirt instance class [mscorlib]System.Collections.Generic.'IEnumerator`1'<!0>
        class [mscorlib]System.Collections.Generic.'IEnumerable`1'<int32>::GetEnumerator()

    stloc enumerator

    .try
    {

    _CONDITION:

        ldloc enumerator
        callvirt instance bool class [mscorlib]System.Collections.Generic.'IEnumerator`1'<int32>::MoveNext()
        brfalse.s _LEAVE

    _STATEMENTS:

        ldloc enumerator
        callvirt instance !0 class [mscorlib]System.Collections.Generic.'IEnumerator`1'<int32>::get_Current()
        call void [mscorlib]System.Console::WriteLine(int32)
        br.s _CONDITION

    _LEAVE:

        leave.s _CONTINUE
    }
    finally
    {
        ldloc enumerator
        callvirt instance void [mscorlib]System.IDisposable::Dispose()

        endfinally
    }

_CONTINUE:

With the comments and labels in place, it should be pretty easy to follow this code. Let’s walk through it quickly. Two local variables are required, one for the list and one for the enumerator. The strange-looking type names are to support generics in MSIL. 'List`1' indicates that the List type with a single type parameter is to be used. This distinguishes between generic types with the same name but with a different number of type parameters. The type list between the angle brackets denotes the actual types to use in runtime specialization. With the local variables defined, the next step is to create the list using the newobj instruction and populate it by calling its Add method. Generics are primarily intended for creating type-safe collections. A good example of this is the code for adding the numbers to the list. The previous example used an ArrayList so the numbers first had to be boxed before adding them to the collection. In this case we can simply push the number on the stack and call the Add method. We just need to make sure to indicate the particular specialization we require. The Add method’s single parameter is !0 which indicates the zero-based type parameter. Now that the collection is ready, we can implement the actual for each loop. To begin enumerating we first get the IEnumerator<T> implementation from the collection and store it in out enumerator local variable. At this point we enter the protected range of instructions, calling the enumerator’s MoveNext method until it returns false. The finally exception handler is then used to call the enumerator’s Dispose method.

Introduction to MSIL – Part 7 – Casts and Conversions

KennyKerr — Tue, 19 Oct 2004 18:43:00 GMT

After a brief hiatus, I decided to post a few more parts of the Introduction to MSIL series.

Casting and type conversion are often raised as topics of concern among programmers. It might be concerns over performance and safety or simply that the implications of casts and conversions are not well understood. In this part of the series I explore these concepts. I will primarily use C++ for illustrations as it is more descriptive in its language constructs related to casting. Of course we will also consider the CLI instructions that are ultimately generated.

Static Casts

When converting from a type with a small value space to a larger value space, an implicit cast can be employed as there is no danger of losing data. This is safe as long as the resulting value space is a superset of the smaller value space. Converting from a 32-bit integer to a 64-bit integer is guaranteed to be accurate, so compilers don’t require any explicit confirmation from the programmer. On the other hand, when converting from a larger type to a smaller type, it is possible for the data to be truncated; therefore compilers typically require confirmation in the form of a cast. This is referred to as a static cast, since only static type information is used in determining the conversion behavior. A static cast is often considered dangerous, as the compiler places the responsibility of ensuring the safety of the cast squarely in the hands of the programmer. Consider the following C++ code:

Int32 small = 123;
Int64 big = small;

small = static_cast<Int32>(big);

Conversion from the small variable to the big variable is implicit but conversion from big to small requires the static_cast operator to avoid a compiler warning regarding possible loss of data. Although static casts can and should be considered dangerous, the compiler goes to great lengths to ensure that it is at least in the realm of possibility that the cast will actually be correct at runtime. It does this by considering the static type information, and in the case of user-defined types, whether there are any conversion operators defined. All this is determined at compile-time. Consider the following representation in MSIL:

.locals init (int32 small,
int64 big)

// Int32 small = 123;
ldc.i4.s 123
stloc small

// Int64 big = small;
ldloc small
conv.i8
stloc big

// small = static_cast<Int32>(big);
ldloc big
conv.i4
stloc small

The ldc.i4.s instruction pushes a 4 byte (32-bit) integer with a value of 123 onto the stack. This is then stored in the small local variable using the stloc instruction. To assign the value of the small variable to the big variable, the value is first pushed onto the stack using the ldloc instruction. The conv.i8 instruction then converts the value to an 8 byte (64-bit) integer. This is then popped off the stack and stored in the big local variable using the stloc instruction. Finally, to convert the value of the big variable to the small variable again, the value stored in the big local variable is pushed onto the stack and the 8 byte value on the stack is converted to a 4 byte value using the conv.i4 instruction. This is then stored in the small local variable using the stloc instruction.

As you can see, MSIL does not make a distinction between implicit and explicit conversions. Everything is explicit. Overflow protection, however, needs to be requested directly, as by default no checking is provided. In the code above, no provision is made for overflow. This code happens to be safe because we know it to be so, but if the value stored in big was too large to be stored in a 4 byte integer, it would simply overflow with no warning at runtime.

Checked Conversions

To detect overflow errors, you can simply replace the conv.i4 instruction in the previous example with the conv.ovf.i4 instruction. If the value on the stack is too large to be represented by the specified type, an OverflowException object is thrown. The C++/CLI language design, introduced in Visual C++ 2005, does not yet provide a language feature to request conversion with overflow checking. A feature is being considered to add a checked keyword, similar to the one provided by C#:

checked
{
    int small = 123;
    long big = small;

    small = (int) big;
}

Any arithmetic operations and conversions in a checked block or expression will include overflow-checking. As mentioned above, the conv.<type> set of instructions are replaced by conv.ovf.<type>. Other instructions that could result in overflow also have corresponding checked versions. For example the add instruction, described in part 2, has corresponding add.ovf and add.ovf.un instructions for checked addition of signed and unsigned values respectively.

Dynamic Casts

Static casts can also be used to cast up a class hierarchy, in effect performing a cast of a polymorphic type but making an assumption that the cast will always succeed at runtime. This is useful to avoid the cost of a run-time type check, assuming it is safe. Of course, this is generally a dangerous assumption to make.

Dynamic casts are used to allow the determination of whether the cast is valid to be deferred to run-time. Of course this only makes sense with polymorphic types. As with static casts, the compiler tries to ensure that it is at least plausible that the cast may succeed. Trying to dynamically cast from an integer to a House object will be caught by the compiler and flagged as an error. Consider the following C++ code. Assume BeachHouse and Townhouse are subclasses of House.

House^ house = gcnew BeachHouse;

if (nullptr != dynamic_cast<BeachHouse^>(house))
{
Console::WriteLine("It's a beach house!");
}

if (nullptr != dynamic_cast<Townhouse^>(house))
{
Console::WriteLine("It's a townhouse!");
}

This code will compile perfectly without the compiler knowing whether the casts will succeed. Both casts require a run-time type check to determine whether the house is in fact a BeachHouse or a Townhouse. dynamic_cast returns a nullptr, the C++ representation for a null pointer or handle, if the cast cannot be performed. Let’s see how we can represent this in MSIL:

.locals init (class House house)

newobj instance void BeachHouse::.ctor()
stloc house

ldloc house
isinst BeachHouse
ldnull

beq.s _TOWN_HOUSE

ldstr "It's a beach house!"
call void [mscorlib]System.Console::WriteLine(string)

_TOWN_HOUSE:

ldloc house
isinst TownHouse
ldnull

beq.s _CONTINUE

ldstr "It's a town house!"
call void [mscorlib]System.Console::WriteLine(string)

_CONTINUE:

Having read through this series thus far, this code should be pretty clear. A BeachHouse object is created and a reference to it is stored as a local variable. The dynamic cast is performed by pushing the reference onto the stack and using the isinst instruction to do the type check. The isinst instruction evaluates whether the reference on the stack is a BeachHouse or a subclass of the BeachHouse type. If it is, then the reference is cast to the type defined in the isinst instruction and pushed onto the stack. Otherwise the value null is pushed onto the stack. The expression for the if statement is constructed by pushing a null reference onto the stack and using the beq.s instruction to transfer control to the target if isinst also pushed null onto the stack. The same type check and conditional branching is done for the Townhouse type, with execution ultimately continuing on.

If your application assumes that the dynamic cast must always succeed for the program to be correct, but you don’t want to resort to a static cast to ensure correctness in the exceptional case, you can employ the castclass instruction in place of the isinst instruction. The castclass instruction performs the same type check, but instead of pushing null onto the stack if the type check fails, it throws an InvalidCastException object. If this is the behavior you are looking for, you can use the safe_cast operator in C++ or a simple (C-style) cast in C#.

Read part 8 now: The for each Statement

Introduction to MSIL – Part 6 – Common Language Constructs

KennyKerr — Thu, 23 Sep 2004 17:04:00 GMT

In parts 1 through 5 of the Introduction to MSIL series we focused on MSIL and the constructs it provides for writing managed code. In the next few sections we are going to examine some common language features that are not intrinsic to instruction-based languages like MSIL. Understanding the intermediate language instructions that are generated for common, programming language statements is critical to acquiring an instinct for the performance characteristics of your code as well as to more easily track down subtle bugs.

In this part we are going to look at some common constructs that are present in many languages. I will be expressing simple examples in C# although the explanations are common to many programming languages that offer the same general constructs. As I am trying to explain a general principle, I may not always present the exact code that a given compiler may produce. The point is to learn more about what compilers generate in general. I encourage you to compare the instructions generated by different compilers using tools like ILDASM.

Programming languages like C and C++ enabled much more rapid development of software compared with classic assembly language programming. A number of rich constructs played a significant part in providing this productivity. Concepts like expressions, selection statements (e.g. if and switch), and iteration statements (e.g. while and for) are what makes these portable languages so powerful. Even more productivity was gained by introducing concepts like objects, but before objects were even considered, expressions and statements provided enormous benefit to the assembly language programmer struggling to build large applications and operating systems. Let’s look at some of these statements.

The if-else Statement

Consider the following simple method that checks that its argument is not null before proceeding.

void Send(string message)
{
    if (null == message)
    {
        throw new ArgumentNullException("message");
    }

    /* impl */
}

The if statement looks innocent enough. If the expression is true, execution enters the scope block and an exception is thrown. If the expression is false, execution jumps to the first statement following the if statement. There’s enough going on here that I will leave the else clause as an exercise for the reader.

Even if you have never used C# before, this code should be pretty clear (assuming you’re a programmer). So how does the compiler turn this innocent little if statement into something the runtime will understand? As with most programming tasks, there are a number of ways to solve the problem.

In many cases it is necessary for a compiler to create temporary objects. Expressions are a common source of temporary objects. In practice, compilers avoid creating temporaries as much as possible. Assuming a non-optimizing compiler you could imagine the compiler turning the code above into something like this.

bool isNull = null == message;

if (isNull)
{
throw new ArgumentNullException("message");
}

This is not to say that an optimizing compiler would not use temporary objects in this manner. Temporary objects can often help to generate more efficient code depending on the circumstance. Here the result of the expression is first evaluated to a boolean value. This value is then passed to the if statement. This is not all that different from the previous example, but after reading the last 5 parts of this series it should be clear why this is much more appealing to an instruction-based language. It’s all about breaking down statements and expressions into simple instructions that can be executed one by one. Let’s consider one implementation of the Send method.

.method void Send(string message)
{
    .maxstack 2

    ldnull
    ldarg message
    ceq

    ldc.i4.0
    ceq

    brtrue.s _CONTINUE

    ldstr "message"
    newobj instance void [mscorlib]System.ArgumentNullException::.ctor(string)
    throw

    _CONTINUE:

    /* impl */

    ret
}

If you’ve been following along with this series you should be able to understand much of this method declaration. Let’s step through it quickly. The temporary object that the compiler might generate silently is made explicit in MSIL, although it is not named and lives very briefly on the stack. (I guess whether it is actually explicit is debatable.) Can you spot it? First we compare message to null. The ceq instruction pops two values off the stack, compares them, and then pushes 1 onto the stack if they are equal or 0 if they are not (this is the temporary). The code may seem overly complicated. The reason is that MSIL does not have a cneq, or compare-not-equal-to, instruction. So we first compare message to null then compare the result to zero, effectively negating the first comparison. More on this in a moment.

Now that we’ve got a handle on the expression result, the brtrue.s instruction provides the conditional branching you would expect from an if statement. It transfers control to the given target if the value on the stack is non-zero, thereby skipping over the if clause’s logical scope block.

Of course there is more than one way to skin a cat. The example above is very similar to what the C# compiler that I am using generates, although it uses an explicit temporary local variable to store the result of the expression. This implementation seems a bit awkward. Ultimately it does not matter too much as the JIT compiler can probably optimize away any differences. Nevertheless it is a useful exercise to see how we can simplify this implementation. The first thing we can attempt is to reduce the number of comparisons. I mentioned that there is no matching not-equal-to version of the ceq instruction. On the other hand there is a branch-on-false version of the brtrue.s instruction. Quite predictably, it is called brfalse.s. Using the brfalse.s instruction completely removes the need for the second comparison.

Finally, as a C++ programmer you would expect the compiler to use the equality operator, since one of the operands is a System.String type which has an equality operator defined for it, and this is exactly what the C++ compiler does.

.method void Send(string message)
{
    .maxstack 2

    ldnull
    ldarg message
    call bool string::op_Equality(string, string)

    brfalse.s _CONTINUE

    ldstr "message"
    newobj instance void [mscorlib]System.ArgumentNullException::.ctor(string)
    throw

    _CONTINUE:

    /* impl */

    ret
}

Ultimately the JIT compiler optimizes the code as best it can, including inlining methods where appropriate. You should not be surprised if these two implementations result in the same machine code at the end of the day.

The for Statement

Before we delve into the implementation of the for statement, commonly referred to as for loops, lets quickly review how the for statement works. If your background is in Visual Basic or even C# you may not be all that familiar with this extremely useful statement. Even C++ programmers are encouraged to avoid the for statement for common iterations in favor of the safer std::for_each algorithm from the Standard C++ Library. There are however, many interesting applications of the for loop that have nothing to do with iterating over a container from beginning to end.

The following simple pseudo code illustrates the construction of a for statement.

for ( initialization expression ; condition expression ; loop BLOCKED EXPRESSION
{
statements
}

The for statement consists of three semi-colon delimited expressions followed by a scope block. (Don’t get me started on for statements without scope blocks.) You can use any expressions you wish as long as the condition expression results in a value that can be interpreted to mean either true or false. The initialization expression is executed first and exactly once. It is typically used to initialize loop indices for iteration or any other variables required by the for statement. The condition expression is executed next and is executed before each subsequent loop. If the expression evaluates to true, the scope block will be entered. If the expression evaluates to false, control is passed to the first statement following the for statement. The loop expression is executed after every iteration. This can be used to increment loop indices or move cursors or anything else that might be appropriate. Following the loop expression, the condition expression is evaluated again and so on and so forth. For a more complete description, please consult your programming language documentation. Here is a simple example that will write the numbers zero through nine to any trace listeners you have set up.

for (int index = 0; 10 != index; ++index)
{
Debug.WriteLine(index);
}

We have already spoken about how an if statement might be implemented in MSIL. Lets now make use of that knowledge to deconstruct the for statement into something simpler to interpret by a computer while still using C#.

    int index = 0;
    goto _CONDITION;

_LOOP:

    ++index;

_CONDITION:

    if (10 != index)
    {
        // for statements
        Debug.WriteLine(index);

        goto _LOOP;
    }

That looks nothing like the for statement! Here I use the infamous goto, more generally referred to as a branch instruction. Branching is inevitable in languages that don’t support selection and iteration statements. It serves no purpose in languages like C++ and C# other than to obfuscate the code. (If you’re a major supporter of the goto statement, please don’t feel the need to share that with me. Feel free to talk about its merits on your own blog.) Code generators however often use goto statements as it is a lot simpler for a code generator to use than to construct the expressions for a for statement for example.

By following the branching in the code example above you should be able to see how we construct the for statement semantics by executing the condition and then jumping up to the loop expression to conditionally loop again. Now let’s consider how we can implement this in MSIL.

    .locals init (int32 index)
    br.s _CONDITION

_LOOP:

    ldc.i4.1
    ldloc index
    add
    stloc index

_CONDITION:

    ldc.i4.s 10
    ldloc index
    beq _CONTINUE

    // for statements
    ldloc index
    box int32
    call void [System]System.Diagnostics.Debug::WriteLine(object)

    br.s _LOOP

_CONTINUE:

After initializing the index local variable we branch to the _CONDITION label. To evaluate the condition I pushed the value 10 onto the stack followed by the index value. The beq, or branch on equal, instruction pops two values off the stack and compares them. If they are equal it transfers control to the _CONTINUE label thereby ending the loop; otherwise control continues through the ‘statements’ in the for loop. To write the index to the debug trace listener I push the index onto the stack, box it and call the static WriteLine method on the System.Diagnostics.Debug reference type from the System assembly. Following this, the br.s instruction is used to transfer control to the loop expression for the next iteration.

Read part 7 now: Casts and Conversions

Introduction to MSIL – Part 5 – Exception Handling

KennyKerr — Wed, 15 Sep 2004 23:58:00 GMT

In this part of the Introduction to MSIL series I will introduce the constructs that the CLI provides for exception handling.

A try block is used to protect a range of instructions. If an exception is thrown by one of the instructions in the protected block, or by any method called directly or indirectly within the protected block, control is transferred to an appropriate exception handler. A try block is declared with the .try directive.

.try
{
/* protected code */

leave.s _CONTINUE
}
<exception handler>

_CONTINUE:

The last instruction in the try block is the leave.s instruction which transfers control to the _CONTINUE label. The leave.s instruction ensures that the evaluation stack is emptied and that the appropriate finally blocks are executed. It follows that if control leaves the protected block by some other means, that an exception was thrown. The exception handler must immediately follow the try block.

The CLI offers four different kinds of handlers that you can employ to construct the error handling semantics that you require for your programming language or application. Let’s examine each one individually.

Catch

The catch exception handler, or catch block, has to be the most well-known form of exception handling, since it is provided directly by C++ as well as C#. A catch block is declared with the catch keyword. The catch clause includes the type of exception that the handler is willing to catch as well as the block of code to transfer control to, given a matching exception object. Catch blocks can also be chained together after a single try block for convenience. The first catch clause with a matching exception type will be chosen as the handler. Consider this example.

.try
{
    ldstr "I'm not a number"
    // ldnull
    // ldstr "123"
    call int32 [mscorlib]System.Int32::Parse(string)

    leave.s _CONTINUE
}
catch [mscorlib]System.ArgumentNullException
{
    callvirt instance string [mscorlib]System.Exception::get_Message()
    call void [mscorlib]System.Console::WriteLine(string)

    leave.s _CONTINUE
}
catch [mscorlib]System.FormatException
{
    callvirt instance string [mscorlib]System.Exception::get_Message()
    call void [mscorlib]System.Console::WriteLine(string)

leave.s _CONTINUE
}

Here we ask the In32::Parse method to parse "I'm not a number", which predictably throws a FormatException. The first catch handler is never given an opportunity to execute. The FormatException handler dutifully writes a message describing the exception to the console and then calls the leave.s instruction to transfer control to the _CONTINUE label. Where was the exception object? The runtime ensure that a reference to the exception is pushed onto the stack before the handler is invoked. You can experiment by commenting out the ldstr instruction in the try block and un-commenting the ldnull instruction. This will push a null reference onto the stack and result in an ArgumentNullException instance being thrown by the Parse method.

Filter

The filter exception handler is a strange construct for a C++ programmer like me. Instead of matching on an exception type, the filter handler provides a scope block where it can evaluate whether it wants to handle the exception. It indicates to the runtime that it wants to handle the exception by pushing a value of 1 onto the stack. If it decides not to handle the exception it pushes a value of 0 onto the stack. Following the “evaluation” block is the handler block that contains the code that will handle the exception.

.try
{
    // ldstr "I'm not a number"
    ldnull
    // ldstr "123"
    call int32 [mscorlib]System.Int32::Parse(string)

    leave.s _CONTINUE
}
filter
{
    ldstr "filter evaluation\n\t"
    call void [mscorlib]System.Console::Write(string)

    callvirt instance string [mscorlib]System.Exception::get_Message()
    call void [mscorlib]System.Console::WriteLine(string)

    ldc.i4.1
    endfilter
}
{
    ldstr "filter handler\n\t"
    call void [mscorlib]System.Console::Write(string)

    callvirt instance string [mscorlib]System.Exception::get_Message()
    call void [mscorlib]System.Console::WriteLine(string)

leave.s _CONTINUE
}

The try block will result in an ArgumentNullException exception because a null reference is pushed onto the stack as an argument to the Int32::Parse method.

The filter is then given a chance to determine whether it wants to handle the exception. In this case it simply writes the error message to the console and then pushes the value 1 onto the stack, using the ldc.i4.1 instruction, to indicate that it wants to handle the exception. The endfilter instruction is called to return from the filter clause.

The filter handler block is then called to handle the exception. Notice that both the evaluation and handler blocks can access the exception in flight by popping it off of the stack. Keep in mind that you can throw any reference type so don’t assume that the object reference you pop off of the stack in the filter handler is an instance of System.Exception. This is not a problem for the catch handler since you will know the type of the exception object.

Finally

The finally exception handler should be recognized by C# programmers and C++ programmers familiar with Structured Exception Handling (SEH). A finally block associated with a try block is always executed, regardless of whether control leaves the protected try block through normal means, using the leave.s instruction, or as a result of an exception, using the throw instruction. It provides a reliable mechanism of ensuring a certain block of code is always run for those languages that do not provide destructors, such as C and C#. Although the C programming language does not target the CLI, SEH is used often enough that the Microsoft C/C++ compiler provides the __finally keyword for the same reason.

.try
{
    /* protected code */

    leave.s _CONTINUE
}
finally
{
    /* cleanup code */

endfinally
}

Fault

The fault exception handler is similar to the finally block except that it is invoked only if it’s associated try block is left as a result of an exception. After the fault handler has been given an opportunity to execute, the exception continues on its way in search or a handler that is willing to catch it.

.try
{
    /* protected code */

    leave.s _CONTINUE
}
fault
{
    /* cleanup code */

endfault
}

With that brief introduction to exception handling out of the way, it is time to change gears a bit and talk about how this all relates to popular programming languages like C# and C++/CLI.

Read part 6 now: Common Language Constructs

Introduction to MSIL – Part 4 – Defining Type Members

KennyKerr — Sun, 12 Sep 2004 05:03:00 GMT

In Part 3 of the MSIL series, I introduced the basic syntax for defining types. Using the .class directive, you can define reference types and value types. Choosing the type attributes correctly, you can exercise complete control over the definition of your type.

.class abstract Kerr.Sample.Object
{
}

In this installment, we are going to explore how to declare members of a type. To simplify the samples, I may omit the class definitions at times and simply focus on defining the members.

Constructors

Let’s begin with initializers, known as constructors in languages like C++ and C#. CLI supports both type initializers and instance initializers. Type initializers are a very handy thing and address many of the multithreading challenges you encounter when implementing a singleton in native C++. A type initializer is run exactly once for a given type and the runtime guarantees that no methods of the type will be accessible before the type initializer completes.

.method static void .cctor()
{
    .maxstack 1

    ldstr ".cctor"
    call void [mscorlib]System.Console::WriteLine(string)

ret
}

.method public void .ctor()
{
    .maxstack 1

    ldarg.0
    call instance void [mscorlib]System.Object::.ctor()

    ldstr ".ctor"
    call void [mscorlib]System.Console::WriteLine(string)

    ret
}

.cctor and .ctor are known as special method names. A type initializer, which is optional, is defined as a static method named .cctor with no return value. Type initializers are called static constructors in the C++/CLI and C# languages.

An instance initializer, more commonly referred to simply as a constructor, initializes an instance of a type. A constructor is defined as an instance method named .ctor also with not return value. A constructor is called when an instance is created using the newobj instruction. Here is an example.

.locals (class TypeName obj)
newobj void TypeName::.ctor()
stloc obj

When executed, it will result in the following written to the console, assuming the constructors defined above.

.cctor
.ctor

The newobj instruction allocates a new instance of the type and initializes all its fields to the type-equivalent of zero. Then it calls the particular constructor, disambiguated by the signature, ensuring that the first (zero-based) argument refers to the newly created instance. Once the constructor completes, an object reference is pushed onto the stack for access by the caller.

Methods

Although the CLI defines constructors and properties in terms of methods (with some extra metadata for properties), in this section we are going to look at methods in the sense of C++ member functions or C# methods. Of course most of what is said of methods, also applies to instance constructors and property getters and setters. Virtually all of the interesting things that you can do with methods are controlled by method attributes. There are some common scenarios programmers expect so lets cover a few of them.

Static methods are defined using the static attribute. Static methods, as you would expect, are associated with a type but not an instance.

.method static void StaticMethod() { /* impl */ }

Instance methods simply use the instance attribute in place of the static attribute. The IL Assembler assumes instance as the default so you rarely need to specify it explicitly for method declarations.

.method void InstanceMethod() { /* impl */ }

The opposite is true when calling methods. The call instruction assumes a static method unless you specify otherwise. Here is an example of calling both methods.

call void TypeName::StaticMethod()

ldloc obj
call instance void TypeName::InstanceMethod()

Remember to push the object reference pointing to your instance onto the stack before calling the instance method.

Virtual function calls are an important part of object-oriented design and the CLI provides a great deal of flexibility in controlling whether the static or dynamic type of the object will be used to service the call, as well as how this behavior can be overridden in subclasses. When I refer to the static and dynamic type in this context I am referring to it in the C++ sense of the static type known at compile time and the dynamic type determined at runtime. This is generally referred to as polymorphism. There are two aspects to the virtual function support that you need to keep in mind when programming in MSIL. The first is how you declare your instance methods to support virtual function invocation and the second is how you call the method. It should also go without saying that static methods are by definition not virtual.

A method is marked virtual by adding the virtual attribute to the type header. Consider the following example.

.class House
{
    .method public virtual void Buy()
    {
        .maxstack 1

        ldstr "House::Buy"
        call void [mscorlib]System.Console::WriteLine(string)

        ret
    }

/* etc */
}

.class TownHouse
    extends House
{
    .method public virtual void Buy()
    {
        .maxstack 1

        ldstr "TownHouse::Buy"
        call void [mscorlib]System.Console::WriteLine(string)

        ret
    }

/* etc */
}

The House type has a virtual method called Buy. The TownHouse type extends House and also has a virtual method with the same name. Because of this, TownHouse::Buy is said to override House::Buy. So how do we tell the runtime which method to pick? Obviously if I have a House instance I would like House::Buy to be called, but if I have a TownHouse instance I would like TownHouse::Buy to be called and, being a virtual method, I want this decision to be made at runtime when the actual type is known. So far I have used the call instruction in a number of examples in this series. The call instruction invokes the specified method and will always call the same method regardless of the dynamic type of the object. The callvirt instruction, on the other hand, allows the runtime to determine the specific virtual method implementation to invoke based on the actual type of the object. Consider the following example.

newobj instance void House::.ctor()
stloc house
newobj instance void TownHouse::.ctor()
stloc townHouse

ldloc house
call instance void House::Buy()

ldloc townHouse
call instance void TownHouse::Buy()

ldloc townHouse
call instance void House::Buy()

ldloc townHouse
callvirt instance void House::Buy()

When executed, it will result in the following written to the console.

House::Buy
TownHouse::Buy
House::Buy
TownHouse::Buy

The first call to the Buy method with the house reference invokes the House::Buy implementation since call is only interested in the static, or compile-time, type. The second call to Buy with the townhouse reference invokes the TownHouse::Buy implementation for the same reason. The third call will once again invoke House::Buy despite the fact that the object reference points to a TownHouse. It should now be clear that using the call instruction implies making a compile-time decision on which method to execute. The final method call uses the callvirt instruction to invoke the virtual method House::Buy and since the object reference actually points to a TownHouse, the TownHouse::Buy method will be executed. To be clear, the runtime is not looking at the type of the local variable you declared but rather the type of the object being referenced. We could have stored a reference to a TownHouse in a House local variable and the TownHouse::Buy method would still have been called.

If you want to declare a virtual method but do not want to override an inherited virtual method with the same name, you can use the newslot attribute on the new virtual method in the subclass. If you consider that virtual method invocation by the runtime is not concerned about method names then you should see how this is possible. Just think of newslot as adding a new virtual function pointer to the vtbl for the given type.

CLI virtual methods are very interesting, especially when you consider how and to what extent they are exposed by C++/CLI and C#. This entry is getting long enough so I’ll save that discussion for another day.

Read part 5 now: Exception Handling

Introduction to MSIL – Part 3 – Defining Types

KennyKerr — Thu, 09 Sep 2004 12:49:00 GMT

In this installment of the MSIL series, I describe how types are defined.

Here is a minimal reference type called House. As I write this, we are looking for a house in British Columbia, so this was the first thing that came to mind.

.class Kerr.RealEstate.House
{
    .method public void .ctor()
    {
        .maxstack 1

        ldarg.0 // push "this" instance onto the stack
        call instance void [mscorlib]System.Object::.ctor()

        ret
    }
}

This is a very simple type. Note that you must declare a constructor for a concrete reference type. Unlike languages like C# and C++, the IL assembler will not generate a constructor for you automatically.

Types are defined using the .class directive followed by a type header. The class keyword is used instead of the more intuitive type for historical reasons. When you read class in MSIL source code, just think type. The type header consists of a number of type attributes followed by the name of the type you are defining. To define the equivalent of a C# static class in MSIL you can write the following.

.class abstract sealed Kerr.RealEstate.MortgageCalculator
{
/* members */
}

abstract and sealed are the type attributes. An abstract type cannot be instantiated and a sealed type cannot have sub-types. There are attributes to control visibility, such as public and private. There are attributes to control field layout, such as auto and sequential. For a complete list of attributes please consult the CLI specification. Many attributes are applied automatically, which can save you a lot of typing. Fortunately these defaults are quite intuitive so you should become familiar with them quickly. As an example, extending the System.ValueType from the mscorlib assembly defines a value type. Since the CLI requires that value types be sealed, the IL assembler will automatically add this attribute for you.

The name of the type in the example above is Kerr.RealEstate.MortgageCalculator. The CLI does not recognize namespaces as a distinct concept. Rather the full type name is always used. The syntax shown above is only supported by the IL Assembler that ships with version 2.0 of the .NET Framework. If you are working with version 1.x then you need to group your namespace members with a .namespace directive, as shown below. Note that this syntax is also supported in version 2.0.

.namespace Kerr.RealEstate
{
    .class abstract sealed MortgageCalculator
    {
        /* members */
    }
}

Following the type name you have the option of specifying the base type. The extend keyword is used for this purpose. If no base type is specified, the IL assembler will add the extend clause to make the type inherit from the System.Object type from the mscorlib assembly, resulting in a reference type. And finally, the type header can provide a list of interfaces that the type and its descendants will implement and satisfy, becoming the interfaces of the type.

.class Kerr.RealEstate.RoomList
    extends [System.Windows.Forms]System.Windows.Forms.ListView
    implements Kerr.IView
{
    /* members */
}

In this example, the Kerr.RealEstate.RoomList type has System.Windows.Forms.ListView, defined in the System.Windows.Forms assembly, as its base type. Keep in mind that the CLI requires that every user-defined type extend exactly one other type. The RoomList type also implements the Kerr.IView interface type.

With this basic introduction to type definitions, you should now be able to start defining more interesting types. To define an interface, simply use the interface attribute in your type header. If you want a value type, known as a struct in C#, simply extend the System.ValueType type from the mscorlib assembly. Now you should be able to see why the .class directive is perhaps not the best name for it.

.class interface Kerr.IView
{
/* members */
}

.class Kerr.RealEstate.HouseData
extends [mscorlib]System.ValueType
{
/* members */
}

Read part 4 now: Defining Type Members

Introduction to MSIL – Part 2 – Using Local Variables

KennyKerr — Tue, 07 Sep 2004 13:34:00 GMT

In part 2 of the Introduction to MSIL series, I will be exploring the use of local variables. Without variables, programs would not be very interesting. To illustrate the use of variables, let’s write a simple program to add numbers together.

In an MSIL method, variables are declared using the .locals directive.

.locals init (int32 first,
int32 second,
int32 result)

This statement declares three local variables for the current method. In this case they all happen to be of type int32, which is a synonym for the System.Int32 type. init specifies that the variables must be initialized to the default values for their respective types. It is also possible to omit the variable names. In that case you would refer to the variables by their zero-based index in the declaration. Of course, using variable names improves readability.

Before we continue, I want to make sure that it is clear how the stack is used explicitly in MSIL. When you want to pass values to an instruction, those values need to be pushed onto the stack. To read those values, the instruction must pop them off the stack. Similarly when calling a method, you need to push the object reference (if any) onto the stack followed by each argument that you wish to pass to the method. In the process of invoking the method, all the arguments as well as the object reference will be popped off the stack. To push a value onto the stack, use the ldloc instruction indicating the variable that holds the value. To pop a value off the stack, use the stloc instruction specifying the variable you wish to store the value in. Also keep in mind that values (based on value types) are stored directly on the stack, but objects (instances of reference types) are not since the CLI does not allow reference types to be allocated on the stack. Rather references to objects are stored on the stack. This is analogous to a native C++ object allocated on the heap with a pointer to it stored on the stack. Keep the stack in your mind as you read this series on MSIL. It should help you understand why values are continually pushed and popped on and off the stack.

The next step is to get the numbers from the user.

ldstr "First number: "
call void [mscorlib]System.Console::Write(string)
call string [mscorlib]System.Console::ReadLine()
call int32 [mscorlib]System.Int32::Parse(string)
stloc first

As I mentioned in Part 1, the ldstr instruction pushes the string onto the stack and the call instruction invokes the Write method, popping its argument off the stack. The next call instruction invokes the ReadLine method which returns a string. The returned string is pushed onto the stack and since it is already there, we simply call the Int32::Parse method which pops the string off the stack and pushes the int32 equivalent on. Note that I am omitting any error handling for the sake of clarity. The stloc instruction then pops the value off the stack and stores it in the local variable named 'first'. Getting the next number from the user works the same way except that the value is stored in the local variable named 'second'.

Now that we have read the two numbers from standard input, it is time to add them up. The add instruction can be used for this purpose.

ldloc first
ldloc second
add
stloc result

The add instruction pops two values off the stack and calculates the sum. To push the values of the local variables onto the stack, we use the ldloc instruction. When the add instruction completes, it pushes the result onto the stack and the program pops the value off the stack and stores it in a variable named 'result' using the stloc instruction.

The final step is to display the result to the user.

ldstr "{0} + {1} = {2}"

ldloc first
box int32

ldloc second
box int32

ldloc result
box int32

call void [mscorlib]System.Console::WriteLine(string, object, object, object)

We use the WriteLine overload that takes a format string followed by three object arguments. Each argument to the WriteLine method must be pushed onto the stack one by one. Since the numbers are stored as int32 value types, we need to box each value; otherwise the method signature won’t match.

The ldloc instruction pushes each argument onto the stack. The box instruction is then used for each int32 argument. Boxing involves popping the value off the stack, constructing a new object containing a copy of the value and then pushing a reference to the object onto the stack.

Here is the complete program.

.method static void main()
{
    .entrypoint
    .maxstack 4

    .locals init (int32 first,
                  int32 second,
                  int32 result)

    ldstr "First number: "
    call void [mscorlib]System.Console::Write(string)
    call string [mscorlib]System.Console::ReadLine()
    call int32 [mscorlib]System.Int32::Parse(string)
    stloc first

    ldstr "Second number: "
    call void [mscorlib]System.Console::Write(string)
    call string [mscorlib]System.Console::ReadLine()
    call int32 [mscorlib]System.Int32::Parse(string)
    stloc second

    ldloc first
    ldloc second
    add
    stloc result

    ldstr "{0} + {1} = {2}"

    ldloc first
    box int32

    ldloc second
    box int32

    ldloc result
    box int32

    call void [mscorlib]System.Console::WriteLine(string, object, object, object)

ret
}

The last thing to notice about this example is that I indicated the method will use at most four stack slots. This is to accommodate the four arguments passed to the WriteLine method at the end of the main method.

Read part 3 now: Defining Types

Introduction to MSIL – Part 1 – Hello World

KennyKerr — Tue, 07 Sep 2004 05:55:00 GMT

When describing C++/CLI and how it relates to C#, I am often tempted to discuss the Microsoft intermediate language (MSIL) that the Visual C++ and Visual C# compilers generate. The trouble is that most programmers are not familiar with MSIL and programs like ILDASM aren’t that helpful to the newbie because the MSIL they display, although painfully correct, is not very readable. For this reason I decided to post a few entries introducing MSIL at a basic level. Since I doubt anybody reading this will actually go and start writing code in MSIL, I will skim over some of the uninteresting details and focus on things like defining types, writing methods, calling instructions and handling exceptions.

Please let me know if you find this helpful.

Let’s start with a simple main (entry point) function that displays a really unique message. Unlike C#, CLI does not have any requirement that a method must belong to a class. The entry point function also does not have to be called main; however I will use this name in this series for simplicity.

.method static void main()
{
.entrypoint
.maxstack 1

ldstr "Hello world!"
call void [mscorlib]System.Console::WriteLine(string)

ret
}

The main method above is called a method definition since both the signature as well as the body of the method is provided. In contrast, when a method signature is provided without a body it is referred to as a method declaration. Method declarations are typically used as call targets (when a method is being called) whereas a method definition provides the actual implementation for a method.

A method definition begins with the .method directive and can be defined at global scope or within a class. The application entry point must be static, meaning an instance is not required to call the method, and that is indicated by the static keyword. Declaring a global method static seems redundant but the ILASM compiler complains if you omit the static keyword in some cases. Think of ‘void main()’ as the signature of the method which, as you would expect, indicates that it does not return a value and takes zero arguments.

The .entrypoint directive signals to the runtime that this method is the entry point for the application. Only one method in the application can have this directive.

The .maxstack directive indicates how many stack slots the method expects to use. For example, adding two numbers together involves pushing both numbers onto the stack and then calling the add instruction which pops both numbers off the stack and pushes the result onto the stack. In that example you will need two stack slots.

The ldstr instruction pushes the string that is passed to the WriteLine method onto the stack. The call instruction invokes the static WriteLine method on the System.Console class from the mscorlib assembly. This is an example of a method declaration. It provides the full signature of the WriteLine method (including the string argument) so that the runtime can determine which overload of the WriteLine method to call.

And finally the ret instruction returns execution to the caller. In the case of the entry point method, this would bring your application to an end.

Read part 2 now: Using Local Variables