Tiger Generics vs. Whidbey Generics

I know, I know, micro-benchmarks are questionable, besides we are talking about early betas, etc. But I just couldn't resist making some small testing. First of all, a generic list of integers in Whidbey C#:

using System;
using System.Collections.Generic;

namespace LogicStudio.Whidbey.Tests
{

  public class GenericValueArrayListTest {

    static void Main()
    {
      DateTime start = DateTime.Now;

      List<int> numbers = new List<int>();

      for (int i = 0; i < 1000000; i++)
      {
        numbers.Add(1962);
      }

      foreach (int i in numbers)
      {
        int number = i;
      }

      DateTime finish = DateTime.Now;

      System.Console.WriteLine(String.Format("Elapsed time: {0}", finish.Subtract(start)));
    }

  }

}

Now, it's equivalent in Tiger Java:

package net.logicstudio.tiger.tests;

import java.util.*;

public class GenericValueArrayListTest {

  public static void main(String[] args) {
    Date start = new Date();

    ArrayList<Integer> numbers = new ArrayList<Integer>();

    for (int i = 0; i < 1000000; i++) {
      numbers.add(1962);
    }

    for (int i : numbers) {
      int number = i;
    }

    Date finish = new Date();

    System.out.printf("Elapsed time: %d ms", finish.getTime() - start.getTime());

  }
}

As you can see, I am just creating a million integers list and then accessing every item. C# needed 70-80 milliseconds to complete the job, Java needed 1,430-1,450 milliseconds. Understandable, as Java generics always use reference types so the integers are being boxed/unboxed inside both loops. As I don't expect to see many value collections in real-life situations, the results are not that significative for me, but what about a collection of employees? First the Whidbey C# version:

namespace LogicStudio.Whidbey.Tests
{

  public class Employee
  {
    int id;
    string name;

    public Employee(int anId, string aName)
    {
      this.id = anId;
      this.name = aName;
    }

    public int Id
    {
      get { return this.id; }
    }

    public string Name
    {
      get { return this.name; }

      set { this.name = value; }
    }
  }

}

using System;
using System.Collections.Generic;

namespace LogicStudio.Whidbey.Tests
{

  public class GenericReferenceArrayListTest {

    static void Main()
    {
      DateTime start = DateTime.Now;

      List<Employee> employees = new List<Employee>();

      Employee emp = new Employee(101, "Smith, Joan");


      for (int i = 0; i < 1000000; i++)
      {
        employees.Add(emp);
      }

      foreach (Employee e in employees)
      {
        Employee currentEmployee = e;
      }

      DateTime finish = DateTime.Now;

      System.Console.WriteLine(String.Format("Elapsed time: {0}", finish.Subtract(start)));
    }

  }

}

Now the Tiger Java version:

package net.logicstudio.tiger.tests;

public class Employee {
  private int id;
  private String name;

  public Employee(int anId, String aName) {
    this.id = anId;
    this.name = aName;
  }

  public int getId() {
    return this.id;
  }

  public String getName() {
    return this.name;
  }

  public void setName(String aName) {
    this.name = aName;
  }
}

package net.logicstudio.tiger.tests;

import java.util.*;

public class GenericReferenceArrayListTest {

  public static void main(String[] args) {
    Date start = new Date();

    ArrayList<Employee> employees = new ArrayList<Employee>();

    Employee emp = new Employee(101, "Smith, Joan");

    for (int i = 0; i < 1000000; i++) {
      employees.add(emp);
    }

    for (Employee e : employees) {
      Employee currentEmployee = e;
    }

    Date finish = new Date();

    System.out.printf("Elapsed time: %d ms", finish.getTime() - start.getTime());

  }
}

This time the C# version took 150 ms and the Java version took 630-650 ms. C# took twice the time with the employees list than with the integers list as a consequence of the compiler use of more efficient representations for value types. Java took less than half the time with the employees list than with the integers list because no boxing/unboxing was needed (Java uses objects in both cases). Anyway and so far, Whidbey is faster than Tiger by a wide margin. In compiling all the examples I used the default switches. Of course, these numbers are totally unscientific so there are no more than a personal divertimento and your results may vary widely.

6 Comments

  • Value type collections are probably more common than you estimate; for instance, most Key types that will be used in in Dictionay&lt;K,V&gt; will be value types.

  • I'm not sure you can publish benchmarks of alpha or beta builds...

  • I note that you used ArrayList&lt;Integer&gt; in the Tiger example and not ArrayList&lt;int&gt;. Is that because ArrayList&lt;int&gt; won't compile under Tiger?

  • Lorenzo: I'm not sure about the legal details. Come to think of it, The Server Side had this kind of trouble with its J2EE vs. .NET benchmark. In any event, this is just some personal numbers tossed out, hardly a benchmark...



    Grant: In Tiger you can't say ArrayList&lt;int&gt; (it's a compiler error) because generics are only made of reference types.

  • This strikes me as a very unfair test for several reasons. The biggest of which is that Tiger is all about HotSpot, which is an adaptive compiler. If you change the code to this:



    &lt;pre&gt;

    import java.util.*;



    public class perf {



    public static void main(String[] args) {



    while (true)

    {

    testit();

    }

    }



    public static void testit()

    {



    Date start = new Date();



    ArrayList&lt;Integer&gt; numbers = new ArrayList&lt;Integer&gt;();



    for (int i = 0; i &lt; 1000000; i++) {

    numbers.add(1962);

    }



    for (int i : numbers) {

    int number = i;

    }



    Date finish = new Date();



    System.out.printf(&quot;Elapsed time: %d ms\n&quot;, finish.getTime() - start.getTime());



    }

    }



    &lt;/pre&gt;



    You will notice the execution time goes down the *second* time you put a million items in the list:



    C:\temp&gt;java -cp . perf

    Elapsed time: 1572 ms

    Elapsed time: 1332 ms



    Then if you throw the -server switch on the command line and let the HotSpot optimizer go to town you get an even better series of numbers:



    C:\temp&gt;java -server -cp . perf

    Elapsed time: 1281 ms

    Elapsed time: 982 ms

    Elapsed time: 911 ms

    Elapsed time: 911 ms

    Elapsed time: 902 ms

    Elapsed time: 901 ms



    In general microbenchmarks are (as you point out) to be avoided. But this particular microbenchmark seems especially slanted.

  • It's great to see you some microbenchmarks showing that we did a good job with List&lt;T&gt;! Thanks.



    As you know, Java's generics solution of using Object everywhere but just sprinkling the syntax in your face isn't a great design, especially if you have collections of value types. (And I agree with Karim - a collection of value types like Points or even Int32's isn't that uncommon).



    But beyond any possible implementation differences in the platform itself, we made some interesting design choices with List&lt;T&gt; itself. ArrayList was a study in sub-optimal design - too many parts of the implementation were virtual. This shows up in weird places, like perf overhead of the enumerator. List&lt;T&gt; is much better in this respect, where we explicitly made some design choices to ensure it was faster.



    You might want to look at Array.Sort&lt;T&gt; and Array.IndexOf&lt;T&gt;, if they're publically available in the builds you have. When you implement IComparer&lt;T&gt; on your types, you should see some interesting perf characteristics here.

Comments have been disabled for this content.