How to Locate GCC Regressions

A regression is a bug that did not exist in a previous release. Problem reports for GCC regressions have a very high priority, and we make every effort to fix them before the next release. Knowing which change caused a regression is valuable information to the developer who is fixing the problem, even if that patch merely exposed an existing bug.

People who are familiar with building GCC but who don't have the knowledge of GCC internals to fix bugs can help a lot by identifying patches that caused regressions to occur. The same techniques can be used to identify the patch that unknowingly fixed a particular bug on the mainline when that bug also exists as a regression on a release branch, allowing someone to port the fix to the branch.

These instructions assume that you are already familiar with building GCC on your platform.

Search strategies

If you've got sufficient disk space available, keep old install tree around for use in finding small windows in which regressions occur. Some people who do this regularly add information to Bugzilla about particular problem reports for regressions.

Before you start your search, verify that you can reproduce the problem with GCC built from the current sources. If not, the bug might have been fixed, or it might not be relevant for your platform, or the failure might only happen with particular options. Next, verify that you get the expected behavior for the start and end dates of the range.

The basic search strategy is to iterate through the following steps while the range is too large to investigate by hand:

• Get GCC sources for that date.
• Build GCC, or specific components that are needed for testing.
• Run the test.
• Based on the outcome of the test, find the midpoint of the new search range.

The first three steps are described below. They can be automated, as can the framework for the binary search. The directory contrib/reghunt in the GCC CVS repository includes scripts to do this work. There are several short cuts that can be used to shorten the elapsed time of the search.

Eventually you'll need to identify the patch and verify that it causes the behavior of the test to change.

There are a variety of problems you might encounter, but many of them are simple to work around.

Get GCC sources

Get a local CVS tree using the cvs instructions. Use a read-only tree that is separate from what you use for development or other testing, since it's easy to end up with files in strange states.

rsync

Using rsync to get a local copy of the GCC CVS repository is highly recommended for regression hunts. You'll be checking out the tree used for the regression search over and over again and won't want to affect access times for other GCC developers who are using the real repository, and it will also be faster for you.

Follow the rsync instructions. The full tree takes a lot of disk space, but it's possible to exclude directories you won't need for your hunt. If you're already using rsync, see the short cuts below for making a smaller copy.

CVS mainline

Check out the GCC CVS tree, specifying the date you want to test. You can keep copies of the various ChangeLog files to compare later when you're ready to identify the patch that caused the regression. For example:

    cat <<EOF > cplog
    #! /bin/sh
    mkdir -p logs/`dirname ${1}`
    cp ${1} logs/${1}.${2}
    EOF
    chmod +x cplog

    DATE="2002-05-01 10:15"
    LOGDATE="`echo ${DATE} | sed 's/[-: ]/_/g'`"
    cvs co -D "${DATE}" gcc > log/${LOG_DATE}.log
    find gcc -name ChangeLog -exec ./cplog {} ${LOG_DATE} \;

Don't keep copies of the ChangeLogs in your CVS tree itself; that will slow down new checkouts. Rather than keeping copies of the files, you can also get differences between ChangeLog files using

    cvs diff -D date1 -D date2 ChangeLog

CVS branches

To get all files for a particular date on the branch, check out the branchpoint and then update with all changes between the branchpoint and the desired date. For example,

    BRANCH="gcc-3_2-branch"
    DATE="2002-09-01"
    rm -rf gcc
    cvs co -r ${BRANCH}point gcc
    cvs up -d -j${BRANCH}point -j${BRANCH}:"${DATE}"

This method works reliably when it starts with a clean checkout. Further updates that use the previous date and a new date sometimes run into problems, with a few files not being updated correctly.

Build GCC

The kind of bug you are investigating will determine what kind of build is required for testing GCC on a particular date. In almost all cases you can do a simple make rather than make bootstrap, provided that you start with a recent version of gcc as the build compiler. When building a full compiler, enable only the language you'll need to test. If you're testing a bug in a library, you'll only need to build that library, provided you've already got a compatible version of the compiler to test it with. If there are dependencies between components, or if you don't know which component(s) affect the bug, you'll need to update and rebuild everything for the language.

If you're chasing bugs that are known to be in cc1plus you can do the following after a normal configure:

    cd objdir
    make all-build-libiberty || true
    make all-libiberty
    make all-libcpp || true
    make all-intl || true
    make all-libbanshee || true
    make configure-gcc || true
    cd gcc
    make cc1plus

This will build libiberty, libcpp, libbanshee, intl and cc1plus (make configure-gcc is required since December 2002, make all-intl since July 2003, make all-libbanshee from May until September 2004, make all-libcpp since May 2004, and make all-build-libiberty since September 2004). Alternatively, you can do

    cd objdir
    make all-gcc TARGET-cc1plus

This works since October 2004. When you have built cc1plus, you can feed your source code snippet to it:

    cc1plus -quiet testcase.ii

Run the test

Assuming that there is a self-contained test for the bug, as there usually is for bugs reported via Bugzilla, write a small script to run it and to report whether it passed or failed. If you're automating your search then the script should tell you whether the next compiler build should use earlier or later GCC sources.

Hints for coming up with a self-contained test is beyond the scope of this document.

Identify the patch

Differences in the ChangeLog files can help you identify files that have changed, although you must keep in mind that CVS checkins are not atomic and ChangeLog diffs between two dates are not always an accurate reflection of changes to other files. If the set of changes is small enough you can guess which patch might have caused the regression and update only the files changed by that patch. Remember to look at all ChangeLogs that might list relevant changes, not just the obvious ones. To get an accurate list of files that changed between two times, do a CVS update to the first date and then a CVS update to the second date, and look for lines in the output from the second update that begin with "U ".

The following CVS commands can help you identify changes from one version of a file to another:

• cvs diff -D date1 -D date2 file
• cvs log -N file
• cvs log -N -d"date1<date2" file
• cvs annotate file

When you've identified the likely patch out of a set of patches between the current low and high dates of the range, test a source tree from just before or just after that patch was added and then add or remove the patch by updating only the affected files. You can do this by identifying the revision of each file when the patch was added and then using ccs update -rrev file to get the desired version of each of those files. Build and test to verify that this patch changes the behavior of the test.

Short cuts

If you've narrowed down the dates sufficiently, it might be faster to give up on the binary search and start doing forward updates by small increments and then incremental builds rather than full builds. Whether this is worthwhile depends on the relative time it takes to update the sources, to do a build from scratch, and to do an incremental build.

Similarly, you can do incremental builds when going forward a small amount from the previous build, and go back to builds in clean object directories when building from earlier sources. When moving forward and doing incremental builds, use contrib/gcc_update rather than cvs co or cvs update.

Before building a compiler after updating the sources, check that the new sources are different from the sources for the current ends of the range. If not, make this new date one of the endpoints without doing the build and running the test. Since CVS checkins are not atomic, you can't rely on ChangeLog differences to reflect actual differences to other files.

When you first use rsync you can exclude directories for components that you know you won't be building. If you're already using a local CVS repository via rsync, you can make a cut-down version of it that leaves out directories you don't need for the regression hunt. This makes cvs operations much quicker, making it worthwhile even if the copy is on the same system. It's particularly useful if you'll want to copy it to a system that is low on available disk space. The following, for example, makes a smaller copy of the repository that can be used for finding C and C++ compile-time problems and takes only half the disk space as the full repository.

    cat <<EOF > rsync_exclude
    --exclude=gcc-cvs/benchmarks
    --exclude=gcc-cvs/boehm-gcc
    --exclude=gcc-cvs/old-gcc
    --exclude=gcc-cvs/wwwdocs
    --exclude=gcc-cvs/gcc/libjava
    --exclude=gcc-cvs/gcc/libstdc++-v3
    --exclude=gcc-cvs/gcc/gcc/ada
    --exclude=gcc-cvs/gcc/gcc/testsuite
    EOF

    tar `cat rsync_exclude` -cf - gcc-cvs | gzip > gcc-cvs.tar.gz

If you have preprocessed source code for your test case, Phil Edwards maintains an automated regression search engine which can help narrow down the range of dates in certain cases. The main development branch can be searched back to pre-3.0 days, and other branches are available with less history. (Note: this web server is not always running.)

Work around problems

If one of the test builds fails, try a date or time slightly earlier or later and see if that works. Usually all files in a patch are checked in at the same time, but if there was a gap you might have hit it.

Sometimes regressions are introduced during a period when bootstraps are broken on the platform, particularly if that platform is not tested regularly. Your best bet here is to find out whether the regression also occurs on a platform where bootstraps were working at that time.

If a regression occurs at the time of a large merge from a branch, search the branch.

If a test causes the compiler or the compiled test program to hang, run it from a csh or tcsh script using limit cputime mm:ss so it will fail if it requires more than the amount of time you specified. The same technique can be used to limit other resources, including memory.

Latent bugs can become apparent due to to small changes in code sizes or data layout. Test failures for these bugs can be intermittent, leading to randomness in a binary search for the patch that introduced the bug. This makes it important to see if the patch resulting from a regression hunt looks as if it's actually related to the bug. For example, if a search on i686-pc-linux-gnu comes up with a change to an unrelated target, you're probably looking for such a bug.

Branch and release dates

If no one has provided a range of dates for when a particular mainline regression appeared, you can narrow the search by knowing in which release it first appeared and then testing the mainline between the branchpoint for that release and the branchpoint for the previous release that does not have the bug. Here are some dates for major CVS milestones.

Please send FSF & GNU inquiries & questions to gnu@gnu.org. There are also other ways to contact the FSF.

These pages are maintained by the GCC team.

For questions related to the use of GCC, please consult these web pages and the GCC manuals. If that fails, the gcc-help@gcc.gnu.org mailing list might help.
Please send comments on these web pages and the development of GCC to our developer mailing list at gcc@gnu.org or gcc@gcc.gnu.org. All of our lists have public archives.

Copyright (C) Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110, USA.

Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.

Last modified 2006-06-21