FxCop is heading in the wrong direction - No testability whatsoever
If you don't know what FxCop is, go look. You'll like it.
In response to my release of FxCopUnit which allows an easier experience testing out FxCop custom rules, David Kean of the FxCop blog posted his thoughts on the matter.
I highly respect the folks doing this work, this is a very powerful tool, but what David posted worries me very much.
From his post:
"Unfortunately, his approach is not going to work in the next version of FxCop/Visual Studio for a couple of reasons.
1) We did some work in Orcas to reduce the visibility of API's that did not need to be public for actually writing custom rules (for example, everything in FxCopCommon.dll is now internal) and
2) We also removed the Reflection 'bridge', that is, the ability to create CCI objects from their Reflection counterparts and vice versa."
(my highlighting).
In the name of of all that is OO
It looks like in the name of all that is pure Object Oriented design, David and his team have decided to remove the little tiny shred of testability and automation that somehow managed to escape their capable design wielding, sealed classing, private membering, non virtualizing, OOP fearing hands.
I'm a big fan of Design for Testability (here's an article I published on the matter) and I feel that giving up some of the "pure" OOP principles for the sake of testability is a very good cause. Personally, if I need to make a choice between enabling a future developer to change running instances in my code but gaining a testable system with unit tests, and hiding everything OOP reasons (security reasons are different - they take precedence) - I'd choose testability every time.
Hiding those testable parts which are still left will make it near impossible to thoroughly test your custom rules in any reliable and automated manner. In fact, it's hard enough to test your FxCop custom rules as it is.
Why aren't many people writing their own custom FxCop rules?
it's actually pretty hard to make them work in FxCop in the first place since there is so much misleading documentation on the subject and many hidden "gotchas" to be dealt with. I've spent at least a day trying to figure out why a custom rule I've made isn't loading into FxCop properly - turns out I misspelled a string somewhere in the constructor, and got no helpful warning whatsoever.
Why is no one testing their custom FxCop rules?
David wrote in his blog post:
"Apart from the things mentioned above, it's a great start and its got us thinking, how many users actually automate the testing of custom rules? Is this something you see as important? Would you like to see how we do this internally?"
My bet is "no one is automating testing Fx rules". In fact, that question can be considered very naive -People simply can't easily test fxcop rules; the barriers are too high for 99% of developers to even start considering testing their rules any other way than debugging. When Sasha tried to test the custom rules he wrote for performance, he had to reflector the hell out of FxCopCmd.exe to understand what's going on, and it's still sort of "just works somehow". It took me massive amounts of digging to get to my "findings" about how one could test FxCop custom rules.
Does that answer your question? A system that does not easily lends itself to testing is a system used to write poorly tested code which results in poor quality code. Kind of ironic when you consider that the whole reason FxCop exists is to help increase code quality. (Perhaps some FxCop custom rules for software testability could be created..)
I had to reflector the FxCop SDK to find out that it's possible to write real unit tests for your Custom FxCop rules, and now I get to find out that we are going to lose this wonderful possibility as well. Why oh why? What on earth could be the reason to remove this wonderful ability? For the sake of the pure OOP principles?
Here's a principle that just as important - quality code. Microsoft does want developers to be able to write quality code using the framework, right? So why cut of the very thin branch that you have (obviously unknowingly) supplied to us? Why is using the FxCop API for testing considered such a horrible hack that you have to remove this ability?
Microsoft would do better to expose even more API's to the FxCop developer.
On that note, I would *love* to see how you guys test your rules.
Suggestions for a more testable FxCop API
- Remove the "Sealed" modifier from most of the classes
- Make many methods virtual by default so that they could be overridden to allow testability and remove external dependencies.
- Make some classes "partial" so that they could be extended.
- Add interfaces and allow access to the factories that create objects to change the output of the factory (a-la dependency injection)
- Allow changing the ProjectDefaults' properties (they are all read only and read from an obscure ProjectDefaults.XML file which I can't find anywhere - and still one of the first rules of a testable design - allow changing configuration at runtime)
- do not remove any existing public APIs that allow testing today.