Unit Testing scalability and performance
I had an interesting chat with
Clemens today. I told him about the Rollback ability using ES and he totally disagreed with me that its a viable option for testing database usage for several reasons:
- If you run tests with LOTS of data - rollback will take a loooong time
- You're introducing "context" into your running code - thus changing is runtime behavior, even if ever so slightly
- If you have components that require new transactions or do not support transactions you can't use this technology
So how does he do it? he restores the database every time. Costly? yes - but less expensive than the other way and the weaknesses just listed won't happen. But more importantly - why is this big difference in opinion happening? why do these rollbacks seem perfectly natural to me and so unnatural and irrelevant to him? Its because we're doing the testing in different ways of thinking. Clemens is testing scalability issues and integration issues in a much more fierce manner than I have ever though about doing. I guess it comes with the territory. Personally - I never needed such tests. What would be required to test load balancing, scalability and perf issues?
- Multi threaded testing
- Repeated - long running, large data volume tests
- lots of interaction in the DB for DALs
My tests look nothing like this. I support the approach that when you test something you test it solely for its own functionality and nothing else. for example - an object that is supposed to behave in a specific way with other objects might be tested using mock objects to verify that it works fine. A DAL will be tested for the simple functionality it has, without multiple threads testing it. I know that it works. If I get a scenario that fails it - *then* I'd test the harder stuff. here are pointers from my testing needs:
- they should run as fast as possible so that you can run them all the time. all of them.
- they should test individual object functionality - often using mock objects and interfaces
- Against the database I will test the simplest functionality until I need something bigger. Scalability testing does enter this field until I *need* scalability. KISS principle.
So - what's the big deal? we're both working on the same direction - on different paths. I do suspect a time will come when I will need to get into hardcore perf and scalability testing. I know Rollback won't help me then. But being able to restore a database easily will.
Hopefully - both of these will be available via attributes in my unit tests.
It's possible even today - with a little effort.