GCC Testing Efforts

This page describes regular efforts to test GCC thoroughly, plus ideas for additional testing by volunteers who have machine cycles to spare. For basic information about running the GCC testsuites, see Installing GCC: Testing. For information about testsuite organization and adding new tests, see Test Suites in the GCC Internals manual and the README files in the test suite directories.

Current efforts

• Phil Edwards runs a build and regression tester referred to as the Autocrasher, which tries to nag people via email when they break things. It pays special attention to C++ binary compatibility. (Note: this web server is not always running.)
• Several people perform regular builds and regression test runs and send their test results to the gcc-testresults mailing list.
• Loren J. Rittle operates daily builds and regression tests with machine time donated by the FreeBSD project.
• Michael Chastain runs the GDB test suite regularly and looks for regressions caused by GCC debug output changes. He tests native i686-pc-linux-gnu once every 4-10 days, and reports to the gcc-testresults and gdb-testers mailing lists.
• Dan Miles and Al Stone implemented scripts to collect all of the e-mail sent to the gcc-testresults mailing list, and store it in a database. Every four hours, the database contents are condensed into a results summary that indicates whether the various versions of the compiler on the various platforms have progressed or regressed since the previous report. By clicking on the thumbnail graphs in the summary, one can see graphs of all reported test results for a particular platform and compiler version over time. Further, all of the original test results e-mailed can be viewed by platform, by compiler version, or individually. Suggestions for improvements or bug reports are always welcome.

Ideas for further testing

• Perform regular builds and testing of current GCC sources that are not already being reported regularly; see Installing GCC: Testing for instructions on submitting test results.
• Build cross compilers and test with simulators as described in How to test GCC on a simulator.
• If your system is beefy enough, do regular builds and test runs with RTL consistency checks enabled. This slows the compiler down by an order of magnitude but has found plenty of bugs in the past.
• Set up an autobuilder (see gcc-regression) to nag people in e-mail when they submit patches that break builds. Ideally we would have one of these for at least each of the primary evaluation platforms listed in the current release criteria, but the more the merrier. Kaveh Ghazi < ghazi@caip.rutgers.edu> suggests that the autobuilders should keep track of regressions in the number of warnings and nag patchers until the new warnings are fixed, just as for testsuite regressions. It's important to have the autobuilders coordinate with each other, to avoid flooding contributors with mail.
• Build and test some or all of the following applications, some of which are part of the GCC release criteria. Links for downloads and for information about the packages are available in the build and test guides. If the instructions are incomplete for your target, update them. Additions to this list (accompanied with build and test guides) are welcome.
  

  Name	Language	Build and test guide
  Blitz++	C++	testing-blitz
  Boost	C++	testing-boost
  FTensor	C++	testing-ftensor
  LAPACK	Fortran 77	testing-lapack
  POOMA	C++	testing-pooma
  Qt	C++	testing-qt

• If the operating system kernel you use is normally compiled with GCC, try building it with the current sources. Make sure it boots. If you're building with relatively stable GCC sources such as a release branch, use the newly built kernel for running further GCC tests (keeping in mind the NO WARRANTY section of the GPL).
• Build and test applications that are important to you; investigate and report any problems you find.
• Build and test packages that are normally available on your platform and for which you have access to source.
• Run benchmarks regularly and report performance regressions.

Please send FSF & GNU inquiries & questions to gnu@gnu.org. There are also other ways to contact the FSF.

These pages are maintained by the GCC team.

For questions related to the use of GCC, please consult these web pages and the GCC manuals. If that fails, the gcc-help@gcc.gnu.org mailing list might help.
Please send comments on these web pages and the development of GCC to our developer mailing list at gcc@gnu.org or gcc@gcc.gnu.org. All of our lists have public archives.

Copyright (C) Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110, USA.

Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.

Last modified 2006-06-21