August 31, 2003

Testing and Breakdowns

Brian Marick has formulated an interesting response to an earlier post, Breakdown, not a problem. Here's what he wrote:

I'm wondering - help me out, Michael - how to fit my talk of test maintenance (within this post) into the terminology of Heidegger and Winograd&Flores. The typical response to an unexpectedly broken test is to fix it in place to match the new behavior. Can I call that circumspective? I prefer a different response, one that makes distinctions: is this one of those tests that should be fixed in place, or is it one that should be moved, or one that should be deleted? Is that attending to a breakdown? And should we expect that a habit of attending to that kind of breakdown would lead to (first) an explicit and (over time) a tacit team understanding of why you do things with tests? And does that mean that handling broken tests would turn back into the fast "flow" of circumspective behavior?

First, a little more elaboration on the notion of Breakdown

Breakdowns are not things that happen. Breakdowns are "speech acts"--in this case declarations--that help us make distinctions.

Consider an example: suppose that I am working on a software development project and, at some point, the bug reports go way up, the bugs get harder to fix, developers are making more mistakes, there is less laughter.

Faced with such a situation, it is not enough to simply acknowledge what is so--that the software is buggy and we're losing control. Rather, we want to disinguish something that is missing and to see that what is missing is impacting our ability to deliver higher quality software.

In looking around, asking questions, talking, we might notice that one of our problems is that we are not finding out about defects until they are reported by testing, or until the customer reports them. This makes it harder to diagnose and repair the defects, not to mention the embarassment that results when customers find our defects (apparently of little concern, however, to many software shops).

In further elaborating the nature of this problem, one distinction we might draw is the distinction Feedback. By drawing the distinction feedback, we invent for ourselves a totally new way of thinking about the problem. We no longer are attached to specific solutions (if only we had more code reviews, if only we had better-trained developers, if only the lead developer had not quit, etc.). In fact, we have no solutions at all. All we have is a different view of the problem, one that might prompt a different set of candidate solutions.

This distinction, feedback, could (and indeed has) lead to a whole host of practices--unit testing, acceptance testing, short and frequent iterations, frequent customer contact, etc. Each practice arises, over time, within the context of other declared breakdowns and distinctions drawn. Many of these practices lead to breakthroughs not just for the current project, but for future projects as well, since they focus on the process rather than the result.

Fast forward in time to Brian's example. You're doing Test-driven development (TDD), when at some point, you write a test for some piece of code, and a previous test fails when that new test is executed. In the face of this situation, one might ask, as Brian does, "is this one of those tests that should be fixed in place, or is it one that should be moved, or one that should be deleted?"


I'm willing to bet that most people would fix the broken code in place and move on. There is nothing at all wrong with this--that is until new problems rear their ugly head.

However, think about it: just because problems have not manifested doesn't mean everything is right. Sometimes we need to make tests to verify whether or not things are really alright. This is, in the broader definition, what testing is all about--to manifest a problem when "everything is right." In a "speech acts"approach, to manifest a problem when "everything is alright" is to bring about a breakdown.


Testing could be understood as a process for generating/declaring breakdowns.

Sometimes breakdowns are generated by forcing a problem to manifest--the classic testing activity in which testers "figure out how to break the code."

At other times, it is done by drawing a distinction--that is inventing a way of looking at things that allows us to see something we didn't see before. This is where I believe the recently evolving practices of testing (exploratory testing, context-driven testing, etc.) might come in. Not only are we testing the code--the artifacts from a sequence of coordinated activities--but we are also testing the assumptions which underlie and (often silently) drive those activities, and thus, effect the result.

In this post, Brian comes up with the distincion "Example-driven testing." This single distinction allows him to differentiate two kinds of unit tests:

(1) tests which drive development
(2) tests which verify that new development does not break older code

By differentiating "Examples" (tests that drive) from "Change Detectors" (tests that verify) Brian essentially problematizes the practice of TDD. To problematize something that previously seemed to work fine is to declare breakdown.

So, with all that having been said, I am with you, Brian. I would prefer activity that "makes distinctions"--in this case, to differentiate tests that should be fixed in place from those that should moved from those that should just be deleted. Note however that without the distinction--i.e. "Example-driven Testing" and the accompanying differentiation between tests that drive and tests that verify--the more down-to-earth practical questsion could not have been posed.

And in response to Brian's other query--"should we expect that a habit of attending to that kind of breakdown would lead to (first) an explicit and (over time) a tacit team understanding of why you do things with tests? "--I would respond enthusiastically with a "Yes."

Because we've learned something, and it is not just navel-gazing.

Posted by mhamman at August 31, 2003 09:40 PM | TrackBack