How to Minimize Test Cases for Bugs

In order for a GCC developer to fix a bug, the bug must be reproducible by means of a self-contained test case. Our bug reporting instructions ask that a bug report include the preprocessed version of the file that triggers the bug. Often this file is very large; there are several reasons for making it as small as possible:

Removal proprietary source may be required to allow publication.
Test cases are more easily reproducible if there are fewer dependencies on target system features, such as system headers.
Smaller test cases make debugging easier.
GCC developers prefer bug reports with small, portable test cases.
Minimized test cases can be added to the GCC test suites.

Most test cases can be reduced to fewer than 30 lines.

Direct approach

Always try a direct approach before resorting to the brute-force method of minimizing a large test case:

Write a small test case based on the code at the location reported in an error message. If the test needs to call a library function (e.g., printf), specify the function's declaration directly rather than including a header file.
For suspected bugs in a GCC runtime library, write a small test case that calls that function, either directly or indirectly, with the arguments that cause it to fail. It's OK to include header files that are part of the GCC release.

Brute force approach

The brute force approach gets easier with experience. The first time it can take an hour or two, but some people report getting it down to 20 minutes or so per test case. It is basically an iterative method to cut down on the size of the test case by deleting chunks of code.

After each attempt to shorten the test case, check whether the bug still is evident. If not, back out to the previous version that demonstrated the bug, and try something else. For this, either use the undo function of your editor, or use frequent backup files; alternatively, you can use #if 0 and #endif.

If you have access to the original sources, you've got the choice of starting either with these or with a preprocessed file. Both methods are described below.

Stripping the original code

Copy the original source file, plus header files that you might need to modify, to a separate location where you can duplicate the failure. Try the following, backing up from the unsuccessful attempts.

Delete all functions that follow the one that triggers the failure.
- If it's in an inline function, remove the inline directive or write a dummy function to call it.
- If it's a template function, provide an explicit instantiation.
Strip as much code as possible from the function that triggers the failure; the remaining code doesn't need to make sense.
- Remove function calls.
- Remove C++ I/O. If you need output to demonstrate the bug, use printf and declare it directly: extern "C" int printf(const char *, ...);.
- Remove unused variables.
- Replace user-defined types (structs, classes) with built-in types (int, double).
- Replace typedefs with primitive types.
Delete functions that come before the function that triggers the failure, or replace their definitions with declarations.
Move includes of system headers before other headers.
Remove #include directives, or replace them with only the code that is needed from the included file to compile the main file.
Replace C++ templates with regular functions and classes, or get rid of as many template arguments as possible.
Simplify class hierarchies: Remove base classes, or use base classes instead of derived ones.
Iterate through the following:
- Remove unused class and struct members.
- Remove definitions of unused data types.
- Remove includes of unneeded header files.

Usually, the file will now be down to fewer than 30 lines, plus includes. Repeat the same steps with remaining included header files. To reduce the number of files you are working on, you can directly include the content of the header file into the source file you are working on.

In the next step, run the compiler with -save-temps to get the preprocessed source file.

Stripping preprocessed sources

The preprocessed file contains lines starting with # that tell the compiler the original file and line number of each line of code. Remove these file and line directives from the preprocessed file so that the compiler will report locations in the preprocessed file, using one of the following:

perl -pi -e 's/^#.*\n//g;' bug.ii
sed '/^#/d' bug.ii > bug2.ii
within vi: :g/^#/d

The preprocessed sources will now consist largely of header files. Follow the same steps as for stripping the original code, with these additional techniques.

There will be large stretches of typedefs from system headers and, for C++, things enclosed in namespaces. Delete large chunks working from the bottom up, e.g. whole extern "C" blocks, or blocks enclosed in namespace XXX {...}.
If you get a new compiler error due to a missing definition for a function or type that you deleted, replace the whole deleted chunk with only the missing declaration, or modify the code to no longer use it.

At this stage, you will be able to delete chunks of hundreds or even thousands of lines at a time and can quickly reduce the preprocessed sources to something small.

Please send FSF & GNU inquiries & questions to gnu@gnu.org. There are also other ways to contact the FSF.

These pages are maintained by the GCC team.

For questions related to the use of GCC, please consult these web pages and the GCC manuals. If that fails, the gcc-help@gcc.gnu.org mailing list might help.
Please send comments on these web pages and the development of GCC to our developer mailing list at gcc@gnu.org or gcc@gcc.gnu.org. All of our lists have public archives.

Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.

Last modified 2006-06-21