One of the problems that we come across as our projects mature is that the number of tests increase and so do test suite times. It seems inevitable ... something like the sun setting in the west and the CPU speed doubling every few years. The obvious problem with this inevitability is that it takes LONGER to test the entire application. Sweeping changes therefore take longer as the feedback loop increases and pairs have to begin context switching during the day ... "Did it get to green yet? No ... where were we?"
This introduces many other problems:
- It is very hard to check in every half hour if your test suite takes 45 minutes to run. Hmmm ... you certainly can't check in without testing first so you have to live with the longer checkin cycles and potentially more merge issues.
- Our tests typically use a local SQL Server database - the same one that the tests running in Visual Studio use when you run a test while developing. This means you can't run adhoc tests as you develop while the test suite is running. This is a showstopper since it means the test suite time is dead for development. You also can't browse the application locally since it is also using the database and chokes as data is changed underneath it.
- Your automated build time is now increasing (due to longer test times) so it takes longer to know if you broke the build and longer to get a build to deploy to a test environment.
Enough with the problems ... how about some solutions? We have brainstormed on lots of different possibilities and have come up with a range of fixes:
- Run performance analysis on the test suite and looks for opportunities to decrease test time. We have begun this process using JetBrains dotTrace with some success. There is still more work to be done and it is hard to find the time as release dates approach. It is important to think of your test code as "production" and therefore it should be refactored and improved on a regular basis.
- Separate the database from the tests and look to use mock objects wherever possible. This is difficult for existing projects where sophisticated database logic is in use in several places and interfaces are not being used to allow easy introduction of mocks.
- Offload the testing to other machines in order to benefits from a parallel system with its own CPU and database server. This has been done with some success by using powerful workstations at remote locations when they are not being used.
Look for more solutions and discussion on this topic in the upcoming months.
How do you solve scalability issues on your TDD projects as they evolve beyond 6 or even 12 months of development?
Jonathan Cogley is the CEO and founder of thycotic, a .NET consulting company and ISV in Washington DC. thycotic has just released Thycotic Secret Server which is a secure web-based solution to both "Where is my Hotmail password?" and "Who has the password for our domain name?". Secret Server is the leader in secret management and sharing within companies and teams.