Unit Testing Perils

Adding unit tests to existing code is nowadays applauded for being progressive, almost borderline pedestrian, for most software development shops. This was not the case when I began writing automated unit tests several years ago. It now feels otherworldly to see greater acceptance of unit testing primarily because I'd greatly toned down my advocacy for it. This shift in attitude includes a (substantial) decrease practicing test driven development (TDD). Unit testing can not simply be applied on blind faith hoping to cure all of one's software ills.

Not long ago, I and other developers on a .NET software project were retroactively adding unit tests (no TDD) for recently produced C# code. Part of the process involved removing any cruft specifically code that was not being called by any other code. The motivation was to help increase code coverage by removing any untouched lines of code. We relied on the static analysis features of the Visual Studio plugin, ReSharper, to reveal these isolated areas of code. In ReSharper parlance, Find Usages handles the work of hunting down any dependencies for symbols and functions.

One of those areas ReSharper indicated that no usages found were for a few simple getter/setter properties of a class. This code was then confidently removed. However, it was later discovered that the removed code did indeed serve a purpose and was providing functionality to one of the GUI screens of the WinForms application. More specifically a few of the columns in a 'DataGridView' control that allowed editing no longer did so. They unexpectedly became read-only.

The GUI screens had no tests. The discovery was made via manual end-to-end testing. This 'DataGridView' control was bound to the properties of the class. It inferred from the properties whether they were getters, setters, or both. The 'Set' accessor of the properties were naively removed since the code we wrote did not seem to call it. The grid control however was binding to it and passing values to and from the properties. No setter accessors now meant the associated columns had become effectively non-editable.

Realizing our mistake we rolled back our original edits for that class and cautiously reviewed all other recent refactorings.

Joel Spolsky once stated that manual testing is all you need in developing quality software and that unit testing provides no notable value . Others taking the inevitable opposing view immediately went on the offensive denouncing his claims. Joel's view is more an over reaction to all the TDD zealots who, whether intentional or not, seem to be deemphasizing the value of old fashion manual testing. Meanwhile, the dissenting voices are manifesting as a fear that their do-no-wrong methodology (and possibly their identity) might possibly amount to nothing. Both views are too extreme and leaning one way or the other can cost you in other areas. You need to continually find and maintain a balance in testing and not become complacent with whatever approach you take.

Yes, integration-style tests might have helped in exercising and validating the correct behaviour in the UI but even that might give you a false sense of security. Manual testing does have its virtues. Without it, you might overlook the human elements of UI functionality, design, and usability. Just like it is an incorrect assumption that no compiler errors means your application is fully functional and ready for end-users.

Also, these were CRUD operations. In my experience they tend to be the least risky part of an application, least likely to be buggy, and the quickest to identify and fix. The cost of writing and maintaining these sorts of unit tests do not warrant their benefit. Is it really worth your time and effort chasing after an unrealistic 100% code coverage? You learn your lesson then you move on (part of continually defining the proper testing balance). Regardless, you should confirm that your seemingly harmless refactoring does not produce unwanted side effects. Unit tests help with that but not by themselves.

Putting the merits of unit testing aside for a moment, the inability to easily detect how the .NET framework is referencing and interacting with my code can be somewhat irritating. This datagrid binding issue is another one of those features in .NET where it does something behind the scenes on your behalf (automagically!) but it is not clear (at least not from a coding perspective) what and how it is doing it. I'd experienced this before when trying to implement paging using NHibernate in an ASP.NET GridView control and it was frustrating. Wait until the havoc WebMatrix, LightSwitch, and friends will unleash on the .NET community.