Testing Details

Code Coverage Analysis

Code coverage testing begins with the instrumentation of the program code, sometimes by a preprocessor, sometimes by an object code modifier, sometimes using a special mode of the compiler or linker, to keep track of all possible code paths in a block of source code and to record, during its execution, which ones were taken. Consider the following somewhat typical C snippet:

1. if (read(s, buf, sizeof buf) == -1)
2. error++;
3. else
4. error = 0;

If the error variable has not been initialized, then the code is buggy, and if line 2 is ever executed then the results of the rest of the program will be undefined. The likelihood of an error in read (and a return value of -1 from it) occurring during normal testing is somewhat low. The way to avoid costly support events from this kind of bug is to make sure that your unit tests exercise every possible code path and that the results are correct in every case. But wait, it gets better. Code paths are combinatorial. In our example above, the error variable may have been initialized earlier -- let's say by a similar code snippet whose predicate (``system call failure'') was false (meaning no error occurred). The following example, which is patently bad code that would not pass any kind of code review anyway, shows how easy it is for simple things to become complicated:

1. if (connect(s, &sa, &sa_len) == -1)
2. error++;
3. else
4. error = 0;
5. if (read(s, buf, sizeof buf) == -1)
6. error++;
7. else
8. error = 0;

There are now four code paths to test:

1.

lines 1-2-5-6.

2.

lines 1-2-5-8.

3.

lines 1-4-5-6.

4.

lines 1-4-5-8.

It's usually impossible to test every possible code path -- there can be hundreds of paths through even a small function of a few dozen lines. And on the other hand, merely ensuring that your unit tests are capable (on successive runs, perhaps) of exercising every line of code is not sufficient. This kind of coverage analysis is not in the tool bag of every software engineer in the field -- and that's why QA is its own specialty.

Regression Tests

Fixing a bug is just not enough. ``Obvious by inspection'' is often a cop-out used to cover the more insidious ``writing the smoking gun test would be difficult.'' OK, so there are many bugs which are obvious by inspection, like division by the constant zero. But to figure out what to fix, one must look at the surrounding code to find out what the author (who was hopefully somebody else) intended. This kind of analysis should be documented as part of the fix, or as part of the comments in the source code, or both. In the more common case, the bug isn't obvious by inspection and the fix will be in a different part of the source code than the place where the program dumped core or otherwise behaved badly. In these cases, a new test should be written which exercises the bad code path (or the bad program state or whatever) and then the fix should be tested against this new unit test. After review and check-in, the new unit test should also be checked in, so that if the same bug is reintroduced later as a side effect of some other change, QA will have some hope of catching it before the customers do.