| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Note: This section describes an experimental feature which will be part of Autoconf in a forthcoming release. Although we believe Autotest is stabilizing, this documentation describes an interface which might change in the future: do not depend upon Autotest without subscribing to the Autoconf mailing lists. |
It is paradoxical that portable projects depend on nonportable tools to run their test suite. Autoconf by itself is the paragon of this problem: although it aims at perfectly portability, up to 2.13, its test suite was using DejaGNU, a rich and complex testing framework, but which is far from being standard on Unix systems. Worse yet, it was likely to be missing on the most fragile platforms, the very platforms that are most likely to torture Autoconf and exhibit deficiencies.
To circumvent this problem many package maintainers have developed their own testing framework, based on simple shell scripts whose sole output are their exit status: the test succeeded, or failed. In addition, most of these tests share some common patterns, what results in lots of duplicated code, tedious maintenance etc.
Following exactly the same reasoning that yielded to the inception of Autoconf, Autotest provides a test suite generation frame work, based on M4 macros, building a portable shell script. The suite itself is equipped with automatic logging and tracing facilities which greatly diminish the interaction with bug reporters, and simple timing reports.
Autoconf itself has been using Autotest for years, and we do attest that it has considerably improved the strength of the test suite, and the quality of bug reports. Other projects are known to use some generation of Autotest, such as Bison, Free Recode, Free Wdiff, GNU Tar, each of them having different needs, what slowly polishes Autotest as a general testing framework.
Nonetheless, compared to DejaGNU, Autotest is inadequate for interactive tool testing, which is probably its main limitation.
| 16.1 Using an Autotest Test Suite | Autotest and the user | |
| 16.2 Writing `testsuite.at' | Autotest macros | |
16.3 Running testsuite Scripts | Running testsuite scripts | |
16.4 Making testsuite Scripts | Using autom4te to create testsuite |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
16.1.1 testsuite Scripts | The concepts of Autotest | |
| 16.1.2 Autotest Logs | Their contents |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts Generating testing or validation suites using Autotest is rather easy.
The whole validation suite is held in a file to be processed through
autom4te, itself using GNU M4 under the scene, to
produce a stand-alone Bourne shell script which then gets distributed.
Neither autom4te nor GNU M4 are not needed anymore at
the installer end.
Each test of the validation suite should be part of some test group. A test group is a sequence of interwoven tests that ought to be executed together, usually because one test in the group creates data files than a later test in the same group needs to read. Complex test groups make later debugging more tedious. It is much better keeping keep only a few tests per test group, and if you can put only one test per test group, this is just ideal.
For all but the simplest packages, some file such as `testsuite.at' does not fully hold all test sources, as these are often easier to maintain in separate files. Each of these separate files holds a single test group, or a sequence of test groups all addressing some common functionality in the package. In such cases, file `testsuite.at' only initializes the whole validation suite, and sometimes do elementary health checking, before listing include statements for all other test files. The special file `package.m4', containing the identification of the package, is automatically included if found.
The validation scripts that Autotest produces are by convention called
testsuite. When run, testsuite executes each test
group in turn, producing only one summary line per test to say if that
particular test succeeded or failed. At end of all tests, summarizing
counters get printed. If any test failed, one debugging script gets
automatically generated for each test group which failed. These
debugging scripts are named `testsuite.nn', where nn is
the sequence number of the test group. In the ideal situation, none of
the tests fail, and consequently, no debugging script is generated out
of validation.
The automatic generation of debugging scripts for failed test has the purpose of easing the chase for bugs.
It often happens in practice that individual tests in the validation
suite need to get information coming out of the configuration process.
Some of this information, common for all validation suites, is provided
through the file `atconfig', automatically created by
AC_CONFIG_TESTDIR. For configuration informations which your
testing environment specifically needs, you might prepare an optional
file named `atlocal.in', instantiated by AC_CONFIG_FILES.
The configuration process produces `atconfig' and `atlocal'
out of these two input files, and these two produced files are
automatically read by the `testsuite' script.
Here is a diagram showing the relationship between files.
Files used in preparing a software package for distribution:
subfile-1.at ->.
... \
subfile-i.at ---->-- testsuite.at -->.
... / \
subfile-n.at ->' >-- autom4te* -->testsuite
/
[package.m4] ->'
|
Files used in configuring a software package:
.--> atconfig
/
[atlocal.in] --> config.status* --<
\
`--> [atlocal]
|
Files created during the test suite execution:
atconfig -->. .--> testsuite.log
\ /
>-- testsuite* --<
/ \
[atlocal] ->' `--> [testsuite.nn*]
|
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
When run, the test suite creates a log file named after itself, e.g., a
test suite named testsuite creates `testsuite.log'. It
contains a lot of information, usually more than maintainers actually
need, but therefore most of the time it contains all that is needed:
A very bad Unix habit which is unfortunately wide spread consists of
setting environment variables before the command, such as in
`CC=my-home-grown-cc ./testsuite'. This results in the test suite
not knowing this change, hence (i) it can't report it to you, and (ii)
it cannot preserve the value of CC for subsequent runs(5). Autoconf faced exactly the same problem, and solved it by asking
users to pass the variable definitions as command line arguments.
Autotest requires this rule too, but has no means to enforce it; the log
then contains a trace of the variables the user changed.
The topmost lines of all the `ChangeLog's found in the source hierarchy. This is especially useful when bugs are reported against development versions of the package, since the version string does not provide sufficient information to know the exact state of the sources the user compiled. Of course this relies on the use of a `ChangeLog'.
Running a test suite in a cross-compile environment is not an easy task, since it would mean having the test suite run on a machine build, while running programs on a machine host. It is much simpler to run both the test suite and the programs on host, but then, from the point of view of the test suite, there remains a single environment, host = build. The log contains relevant information on the state of the build machine, including some important environment variables.
The absolute path and answers to `--version' of the tested
programs (see Writing `testsuite.at', AT_TESTED).
The contents of `config.log', as created by configure,
are appended. It contains the configuration flags and a detailed report
on the configuration itself.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The `testsuite.at' is a Bourne shell script making use of special
Autotest M4 macros. It often contains a call to AT_INIT nears
its beginning followed by one call to m4_include per source file
for tests. Each such included file, or the remainder of
`testsuite.at' if include files are not used, contain a sequence of
test groups. Each test group begins with one call to AT_SETUP,
it contains an arbitrary number of shell commands or calls to
AT_CHECK, and it completes with one call to AT_CLEANUP.
Initialize Autotest. Giving a name to the test suite is encouraged if your package includes several test suites. In any case, the test suite always displays the package name and version. It also inherits the package bug report address.
Log the path and answer to `--version' of each program in space-separated list executables. Several invocations register new executables, in other words, don't fear registering one program several times.
Autotest test suites rely on the PATH to find the tested program.
This saves from generating the absolute paths to the various tools, and
makes it possible to test installed programs. Therefore, knowing what
programs are being exercised is crucial to understand some problems in
the test suite itself, or its occasional misuses. It is a good idea to
also subscribe foreign programs you depend upon, to ease incompatibility
diagnostics.
This macro starts a group of related tests, all to be executed in the same subshell. It accepts a single argument, which holds a few words (no more than about 30 or 40 characters) quickly describing the purpose of the test group being started.
Associate the space-separated list of keywords to the enclosing
test group. This makes it possible to run "slices" of the test suite.
For instance if some of your test groups exercise some `foo'
feature, then using `AT_KEYWORDS(foo)' lets you run
`./testsuite -k foo' to run exclusively these test groups. The
title of the test group is automatically recorded to
AT_KEYWORDS.
Several invocations within a test group accumulate new keywords. In other words, don't fear registering several times the same keyword in a test group.
End the current test group.
Initialize an input data file with given contents. Of course, the contents have to be properly quoted between square brackets to protect against included commas or spurious M4 expansion. The contents ought to end with an end of line.
Execute a test by performing given shell commands. These commands should normally exit with status, while producing expected stdout and stderr contents. If commands exit with status 77, then the whole test group is skipped.
The commands must not redirect the standard output, nor the standard error.
If status, or stdout, or stderr is `ignore', then the corresponding value is not checked.
The special value `expout' for stdout means the expected output of the commands is the content of the file `expout'. If stdout is `stdout', then the standard output of the commands is available for further tests in the file `stdout'. Similarly for stderr with `expout' and `stderr'.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts Autotest test suites support the following arguments:
Display the list of options and exit successfully.
Display the version of the test suite and exit successfully.
Remove all the files the test suite might have created and exit. Meant
for clean Makefile targets.
List all the tests (or only the selection), including their possible keywords.
By default all the tests are performed (or described with `--list') in the default environment first silently, then verbosely, but the environment, set of tests, and verbosity level can be tuned:
Set the environment variable to value. Do not run `FOO=foo ./testsuite' as debugging scripts would then run in a different environment.
The variable AUTOTEST_PATH specifies the testing path to prepend
to PATH. It handles specially relative paths (not starting with
`/'): they are considered to be relative to the top level of the
package being built. All the directories are made absolute, first
starting from the top level build tree, then from the
source tree. For instance `./testsuite
AUTOTEST_PATH=tests:bin' for a `/src/foo-1.0' source package built
in `/tmp/foo' results in `/tmp/foo/tests:/tmp/foo/bin' and
then `/src/foo-1.0/tests:/src/foo-1.0/bin' being prepended to
PATH.
Add the corresponding test groups, with obvious semantics, to the selection.
Add to the selection the test groups which title or keywords (arguments
to AT_SETUP or AT_KEYWORDS) match all the keywords
of the comma separated list keywords.
Running `./testsuite -k autoupdate,FUNC' will select all the tests tagged with `autoupdate' and `FUNC' (as in `AC_CHECK_FUNC', `AC_FUNC_FNMATCH' etc.) while `./testsuite -k autoupdate -k FUNC' runs all the tests tagged with `autoupdate' or `FUNC'.
If any test fails, immediately abort testing. It implies `--debug': post test group clean up, debugging script generation, and logging are inhibited. This option is meant for the full test suite, it is not really useful for generated debugging scripts.
Force more verbosity in the detailed output of what is being done. This is the default for debugging scripts.
Do not remove the files after a test group was performed --but they are still removed before, therefore using this option is sane when running several test groups. Do not create debugging scripts. Do not log (in order to preserve supposedly existing full log file). This is the default for debugging scripts.
Trigger shell tracing of the test groups.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts For putting Autotest into movement, you need some configuration and Makefile machinery. We recommend, at least if your package uses deep or shallow hierarchies, that you use `tests/' as the name of the directory holding all your tests and their `Makefile'. Here is a check list of things to do.
Make sure to create the file `package.m4', which defines the
identity of the package. It must define AT_PACKAGE_STRING, the
full signature of the package, and AT_PACKAGE_BUGREPORT, the
address to which bug reports should be sent. For sake of completeness,
we suggest that you also define AT_PACKAGE_NAME,
AT_PACKAGE_TARNAME, and AT_PACKAGE_VERSION.
See section Initializing configure, for a description of these variables. We
suggest the following Makefile excerpt:
$(srcdir)/package.m4: $(top_srcdir)/configure.ac
{ \
echo '# Signature of the current package.'; \
echo 'm4_define([AT_PACKAGE_NAME], [@PACKAGE_NAME@])'; \
echo 'm4_define([AT_PACKAGE_TARNAME], [@PACKAGE_TARNAME@])'; \
echo 'm4_define([AT_PACKAGE_VERSION], [@PACKAGE_VERSION@])'; \
echo 'm4_define([AT_PACKAGE_STRING], [@PACKAGE_STRING@])'; \
echo 'm4_define([AT_PACKAGE_BUGREPORT], [@PACKAGE_BUGREPORT@])'; \
} >$(srcdir)/package.m4
|
Be sure to distribute `package.m4' and to put it into the source hierarchy: the test suite ought to be shipped!
Invoke AC_CONFIG_TESTDIR.
An Autotest test suite is to be configured in directory. This
macro requires the instantiation of `directory/atconfig' from
`directory/atconfig.in', and sets the default
AUTOTEST_PATH to test-path (see section Running testsuite Scripts).
Still within `configure.ac', as appropriate, ensure that some
AC_CONFIG_FILES command includes substitution for
`tests/atlocal'.
The `tests/Makefile.in' should be modified so the validation in your package is triggered by `make check'. An example is provided below.
With Automake, here is a minimal example about how to link `make check' with a validation suite.
EXTRA_DIST = testsuite.at testsuite
TESTSUITE = $(srcdir)/testsuite
check-local: atconfig atlocal $(TESTSUITE)
$(SHELL) $(TESTSUITE)
AUTOTEST = $(AUTOM4TE) --language=autotest
$(TESTSUITE): $(srcdir)/testsuite.at
$(AUTOTEST) -I $(srcdir) $@.at -o $@.tmp
mv $@.tmp $@
|
You might want to list explicitly the dependencies, i.e., the list of the files `testsuite.at' includes.
With strict Autoconf, you might need to add lines inspired from the following:
subdir = tests
atconfig: $(top_builddir)/config.status
cd $(top_builddir) && \
$(SHELL) ./config.status $(subdir)/$@
atlocal: $(srcdir)/atlocal.in $(top_builddir)/config.status
cd $(top_builddir) && \
$(SHELL) ./config.status $(subdir)/$@
|
and manage to have `atconfig.in' and $(EXTRA_DIST)
distributed.
| [ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This document was generated by John Paul Wallington on October, 29 2003 using texi2html 1.67.