Test Driven Development with C++: Much harder than it should be

(BTW: Does anyone have any knowledge of the C++ testing frameworks listed here at the bottom of the page? Got any favorite?)

One of the things I'm doing right now in Magen is helping a company that uses hard core C++ for various networking and conferencing software move to a more agile oriented development regime. As part of that they are learning Test Driven Development - in C++. There are a few things I can say about this. Mainly that it's a struggle all the way through.

The fact is that I've never been a C++ wonk and so I'm having some difficulty adjusting to the way things are done in in this language. Add to that that:

  • CPPUnit is the framework of choice for C++ (at least that I know of).
  • There are no refactoring tools for C++
  • Visual C++ 6.0 is horrendous to work with. Lack of usability is apparent in everything.

Some of the early conclusions so far:

  • Creating unit tests in C++ is done by inheritance (no attributes to speak of) and by using special macros to manually add your test cases and test fixtures into the unit test runner
  • The only way we could figure out so far on how to run the unit tests is by manually using the test runner objects from within the main method (which means we are bound to using a console application)
  • Documentation on how to make a project that uses the GUI test runner is practically non-existent. Only a small sample comes with CPPUnit and it's hard to decipher.
  • It pretty much works the same way like NUnit but you have much less
    • types of assertions
    • "attributes" for example, no such thing as testFixtureSetup or TestFixtureTearDown methods to override from the base class
    • ease of use - at least 4 steps needed to create a new unit test

So overall this leads to some facts:

  • If its harder to write tests in C++, less people are likely to do it.
  • If it's harder to refactor code, less people will do it (this is more an issue with the tools - VC++ 6.0)
  • Agile wok in C++ is much harder but I can honestly say it does not have to be.

Sadly it doesn't look like there's any sort of agile innovation in this territory. CPPUnit is not getting any new features soon, and something like the NUnit-Addin in C++ is nothing but a dream. That sucks because there is a lot of code out there being written in C++. Has this suddenly become some sort of "second class" platform for agile tools? If so it could have some sort of explanation. Mainly that could be that when going agile you want as much productivity as you possibly cam and so automatically turn to RAD-oriented tools and IDEs. C++ has never been considered as such. But that could change.

If an Add-in like Visual -Assist is so smart, it could easily add refactoring features to the platform. CPPUnit and variants cold be written to more easily run form a GUI. More and better documentation and blogs could be written on these issues.

And to finish off, here's something I found on my search for a Mock object framework for C++ .

This guy tells it like it is and one could easily replace "Java" with ".Net":

"What happened to the joy of C++? And wasn't the moo book a gas? Clearly I enjoy the stuff. But now I hate C++. The explanation is easy: tools.

After a solid year of Java at its finest, returning to C++ and no good unit test libraries (CPPUnit being about it), no mock objects, no IDEA or decent Eclipse support, no ant or maven, no good coverage tools, no good code analysis, no good dependency analysis on and on and on. Tools make the programmer. And C++ has dog food for tools compared to the wealth of open-source projects for Java. Now that I'm working on a C++ project, my development efforts are at least doubled for the same work, and I continually feel that I am leaving imporant parts of best practices out of the picture. Nor does it look like that's about to change anytime."

[update: looks like people are actually looking for Agile C++ developers. I bet those are really hard to find these days]
Published Thursday, December 02, 2004 6:50 PM by RoyOsherove
Filed under:

Comments

Thursday, December 02, 2004 12:15 PM by Paul Bartlett

# Is "agile C++" an oxymoron

Here are my thoughts on the subject from a while back...

http://blogs.geekdojo.net/pdbartlett/archive/2004/03/09/1339.aspx
Thursday, December 02, 2004 1:41 PM by Kyle Alons

# re: Test Driven Development with C++: Much harder than it should be

Here's one other C++ unit testing framework (haven't used any of them so don't know which is best):
http://cpptest.sourceforge.net/
Thursday, December 02, 2004 1:51 PM by Roy Osherove

# re: Test Driven Development with C++: Much harder than it should be

Hey Kyle, thanks! I've checked out Kinook's Visual Build software a while ago and have often compared it with FinalBuilder which I just love. Just wanted to give props to your company for making a powerful product. I just wish it would be more usable and less "macro" oriented.
Thursday, December 02, 2004 7:25 PM by Brian Button

# re: Test Driven Development with C++: Much harder than it should be

Roy,

check out Ron Jeffries site, http://www.xprogramming.com/software.htm. There are a bunch of links there to C++ testing frameworks. Also check out CppUnitLite from Michael Feathers. I've used this on a project before, and it worked really well. It's available on the objectmentor.com site someplace.

bab


Friday, December 03, 2004 4:04 AM by Dee Zsombor

# re: Test Driven Development with C++: Much harder than it should be

Well Roy you are right it is more difficult than it should be, but not nearly as difficult as you have describe it! Some of the limitations come from the nativeness of the language itself, for example without reflection you cant have a framework that auto discoverers test functions. Also you cant have mock objects without reflection.

As for refactoring support, there is one tool that I can think of: Xrefactory that integrates well in Emacs, Xemacs and I believe Jedit too. Also there is plan to implement DOM parser in CDT (that has decent support) thus you be able to use Eclipse for refactoring Not that I would need another IDE as I'm an (X)Emacs guy. By the way I also think the Visual Studio 6.0 sucks but there are better options anyway.


There is no need to register testcases manually (although I prefer to do this myself for reasons not presented here): there you can use CPPUNIT_TEST_SUITE_REGISTRATION macro, or just configure your editor template support to add these lines automatically. Also you can very well use fancy GUI runners, as far as I know there is one implemented for windows with MFC and another cross-platform with QT. But then again you-will need command line as for automated continuous builds. Supplied GUI runner example is minimalistic enough for you to use one for your self should you need it.

True that you don't have ant, but you have make, nmake, jam, A-A-P, SCons and lot more of build tools that don't need hairy XML syntax to setup a simple build. As for coverage there is Rational Pure Coverage, tcov, gcov, Dynamic Code Coverage, CTC++, and whole lot more. As for dependency analysis your statement is simply not fare.
Saturday, December 04, 2004 9:36 AM by Eric Vautier

# re: Test Driven Development with C++: Much harder than it should be

Arnold Ewald mentions he has implemented a Mock Objects framework for C++.
Saturday, December 04, 2004 9:57 AM by Thibaut Barrère

# re: Test Driven Development with C++: Much harder than it should be

>(BTW: Does anyone have any knowledge of the C++ testing frameworks listed here >at the bottom of the page? Got any favorite?)

My favourite C++ testing framework is TUT (which is not listed!), although it won't work with vc 6 nor vs.net 2002 (it requires a strongly template compliant compiler). It works fine with vc 7.1 (vs.net 2003) and as well with other platforms (solaris etc)

It only requires two header files (no lib!) and I find it very easy to setup and use, very lightweight, and its integration in the build cycle is simple.

I posted about it : http://blogs.dotnetguru.org/tbarrere/?title=unit_testing_with_tut&more=1&c=1&tb=1&pb=1

You can download TUT here : http://tut-framework.sourceforge.net/
Sunday, December 05, 2004 3:08 PM by Reginald Braithwaite-Lee

# Agile C++ Development Shop

I'm the development manager seeking agile C++ developers in Toronto.

I wouldn't say that Test Driven Development ("TDD") with C++ is much harder than it should be (although that's true). I would say:

1. Test-driven development of any applications written 5-15 years ago is hard. TDD was not common, and the architectures of legacy applications are not supportive. Even with great tools, it is hard to use TDD on legacy applications.

The connection with C++ is fairly obvious: much of the C++ code out there was first written 5+ years ago.

As a knock-on effect of this, most of the C++ developers kicking around did not employ TDD during their "formative years." As a result, they don't use TDD tools. Try making a TDD C++ tool: you'll find it very hard to get C++ developers to adopt it.

2. Test-driven development of highy OS-initimate applications is hard. Operating system APIs simply aren't written to facilitate test-driven development. That's why WinRunner sucks as a test tool: it's the OS that makes it hard.

Of course, if people were using C++ for three-tier business applications we would find TDD easier. But of course, people aren't writing those applications in C++. People are writing things that are fairly intimate with APIs, and most of these don't facilitate TDD>

Can we factor UIs out and take other steps to make C++ applications easier to test using automation? Of course we can, and we do. But it's inherently harder to do so.

*****

In summary, my experience is that there is a vicious cycle at work: legacy coders (5+ or 10+ years of experience) aren't big on TDD, so the market for TDD is smaller on platforms/languages that have been around longer. This drives fewer tools, which in turn drives lower adoption.

Simultaneously, it's hard work to make API-intimate applications testable. This exerts serious negative pressure on TDD.

p.s. "Hard work"??? I know, just design well from the get-go. This is easy in theory, difficult in practice when the bug list is a mile long, there's a hard cap on a release date next Friday, and the VP of Product Management is drilling down on every developer-hour invested. The "hard work" is often political :-)
Sunday, December 05, 2004 3:14 PM by Reginald Braithwaite-Lee

# re: Test Driven Development with C++: Much harder than it should be

p.p.s. Despite my complaining, we're still hard core about agile development and test-driven development. And yes, I do the "hard work" of selling management on the ROI. For example, later this (12/2004) month, we have a consultant coming in to lead a two day CPPUnit workshop.

Although this seems like an easy thing to arrange, I assure you there has been serious due diligence on investing two days for each developer in this training as opposed to doing other work.

It's easy to say "we'll make up the lost time in improved productivity," but in a professional environment you had better be prepared to say "if we don't do the training here is the schedule for what will be developed by when. If we go ahead with the training here are the specific and measurable improvements in dates and/or scope. And here's the objective analysis backing everything up."