Bugs and Missing Features with Team System Unit Testing: Part I
Here are a couple more examples on the immaturity of the current Team System Unit Testing in "Quality Tools" . Both of these were submitted to the product feedback center, which, unfortunately, only seems to have been catered Towards the pre-release versions of VS.NET. But I still added them anyway, just so that maybe someone looks over at them. Feel free to click on the title links and vote for them if you think they are important enough.
Both of these,BTW, apply to the situation in which you'd like to test than an Exception is indeed thrown from your code when expected.
This one is a bug:
While writing a unit test with an "ExpectedException" attribute and also setting the "Message" property on the excpected exception, the test passes even though the exception that was thrown has a different message.
| Steps to Reproduce: | [TestMethod]
[ExpectedException(typeof(InvalidOperationException),"Some Message")]
public void MyTest()
{
//this test will pass, but it should fail on the exception message being wrong
throw new InvalidOperationException("Some Other Message");
} |
How this one went by into the release is a mystery to me. Obviously no onw has ever used this functionality to actually talk about the fact that it's broken (and no one actually tested for it). Bad.
Here's the second one - it's a suggestion on a missing feature which should have been there.
Example:
[TestMethod]
[ExpectedException(typeof(Exception),"Some Message")]
public void MyTest()
{
//this test will NOT COMPILE claiming that the attribute requires something that DERIVES from Exception
...
}
[TestMethod]
[ExpectedException(typeof(InvalidOperationException),"Some Message")]
public void MyTest()
{
//this test WILL COMPILE
...
}
Right now if I want to write a test than expects an exception, I have to make sure that the thrown exception *has* to derive from Exception, and cannot use the Generic Exception object.
In many cases, developers would like to use a generic exception, either because they are not sure what exception the code will throw, or they just want to use the simplest thing that works.
The current situation makes the developer find some other generic exception, against their better judgment.
This is different than the default behavior in NUnit and I'm curious as to why this is so.
Here are some more observations:
The Test List Window is not Usable for Test Driven Development:
- When the tests are running, you get to see a list of *all the tests*. Passing and failing together. If I'm working with Test Driven Development, the last test is usually the one to fail, but that last test is also usually located waaay down then on the bottom of the list, making me scroll to reach it every time. It's be OK if I could at least sort the list based on pass/fail status, but I can't!.
- Workaround: Use TestDriven.NET. It only shows you the failed tests.
- If a test fails, double clicking on it opens the test failure description dialog page, instead of the failing code. I guess that's OK if that was a regular test, but for unit tests, the default behavior should at least be configurable to jump to the code of the test. Not only that: for an exception, it should put me exactly where the exception occurred (just like double clicking on a call stack line in the output window) .
- Workaround: Use TestDriven.NET: it allows clicking on a line of an exception or error in call stack, and it puts you right on that line in the code editor.
- Running the tests does not necessarily invoke the *build* process. so in Test Driven Development, I have to continuously Build (Ctrl-Shift-B), and then run the current test project (Ctrl-Shift-X). Unacceptable, and should at least be configurable.
- Workaround: Use TestDriven.NET : running tests automatically invokes the build first
- No default hierarchical view of the tests by namespace/class. If you're trying to build an NUnit killer, at least build in the correct UI like NUnit inside the IDE. Right now, if I want to run only some of the test, I have to manually arrange them by class, or namespace, and create my own test lists. That sucks, I'd like a default arrangement that I can use ad-hoc.
- Workaround: Use the latest version of NUnit GUI. It allows running Team System Unit Tests these days too. Just point it at your Team System Dll with unit tests and your'e good to go.