Eddie Garmon's Weblog

some architecture, some c#

Why a Timed Test in NUnit 2.2?

In response to questions about the value add of Timed Tests in NUnit:

There are several uses for timed tests, most of which fall under the umbrella of performance testing.

1) You have a piece of code/hardware that you want certain functions to complete in less than a specified amount of time. If you already have a NUnit test that exercises this functionality, all you have to do is change your [Test] attribute to a [TimedTest(time limit)] attribute an now your unit test plays 2 roles, functional unit test, and performance test.

2) Performance optimization regression testing. Say you have a segment of code that has been optimized, and has a timed test that verifies execution in less than a specified amount of time. When you run this unit test as you add to/refactor your code base, you will immediately be notified if you introduce code that has less performance than your tests allow.

3) Quickness in creating new performance tests. This is as simple as adding a new NUnit test fixture, and adding a timed test.

4) Integration of performance tests with unit tests. If you know how to write one, you know how to write the other. You don't need to integrate another testing framework to do basic performance monitoring.

5) Time limit on long running tests. Once a test times out, execution of the test stops, guaranteeing that a long running test will not slow up the build/test/deployment procedure that you have in place.

6) Add a category to all of your timed tests and with-in NUnit, you have a predefined set of performance tests to run, without checking boxes.

Also of note: I made a change to the code so that only the actual test is timed, not the setup and tear down as well. Get the updates here.

Comments

Jouni Heikniemi said:

In your experience, do you run these performance tests on centralized test servers only, i.e. you don't require developers to run them on their own boxes? I could imagine the varying load conditions (and hardware) on workstations to be a source of random fluctuation in NUnit results if some perf tests failed at times and then again passed on the next run, with no modifications inbetween.

Of course, NUnit and your extension are no worse than any other performance testing technique in this respect, but integrating the perf tests with the unit tests could potentially make the perf-related problems a lot more frequent. What are your thoughts on this?
# February 14, 2005 1:19 AM

Eddie Garmon said:

When we use these tests, it is for improved comfort level. As you said, the better the hardware, the more likely it is to pass the timed test. Currently our developer hardware match or exceed the server hardware that we use.

On the other hand, if the code only passes the timed test because it is running on a Cray, to me suggests the raising of some flags...
# February 14, 2005 1:27 AM

bigjuju said:

doesn't sound like a unit test to me. i suppose if you had an extremely localized algorithm that you could test, that would be acceptable. platform dependent tests tend to be integration/acceptance tests, no?

some testing is better than none.
# February 14, 2005 3:43 PM

Eddie Garmon said:

bigjuju - The times test is not strictly for performance testing. It is for timing your unit tests to validate that they execute in some established period of time.

The timed test does not measure how long the test takes to execute, it only guarantees a test failure and exiting of the test if the specified time elapses before the test completes.

Integration/acceptance tests tend to be system/application wide tests on top of all the unit tests.

As for platform dependence, there should be none. The timed test time out should cover the minimum time required on the slowest supported hardware. Remember, this is performance from the view that it gets done on time, not that the code is the quickest solution possible.
# February 14, 2005 4:35 PM

Why a Timed Test in NUnit 2.2? said:

Pingback from  Why a Timed Test in NUnit 2.2?

# November 27, 2007 12:14 PM
Leave a Comment

(required) 

(required) 

(optional)

(required)