Extending DejaGnu

Adding A New Testsuite

The testsuite for a new tool should always be located in that tools source directory. DejaGnu require the directory be named testsuite. Under this directory, the test cases go in a subdirectory whose name begins with the tool name. For example, for a tool named flubber, each subdirectory containing testsuites must start with "flubber.".

Adding A New Tool

In general, the best way to learn how to write (code or even prose) is to read something similar. This principle applies to test cases and to testsuites. Unfortunately, well-established testsuites have a way of developing their own conventions: as test writers become more experienced with DejaGnu and with Tcl, they accumulate more utilities, and take advantage of more and more features of Expect and Tcl in general.

Inspecting such established testsuites may make the prospect of creating an entirely new testsuite appear overwhelming. Nevertheless, it is quite straightforward to get a new testsuite going.

There is one testsuite that is guaranteed not to grow more elaborate over time: both it and the tool it tests were created expressly to illustrate what it takes to get started with DejaGnu. The example/ directory of the DejaGnu distribution contains both an interactive tool called calc, and a testsuite for it. Reading this testsuite, and experimenting with it, is a good way to supplement the information in this section. (Thanks to Robert Lupton for creating calc and its testsuite---and also the first version of this section of the manual!)

To help orient you further in this task, here is an outline of the steps to begin building a testsuite for a program example.

• Create or select a directory to contain your new collection of tests. Change into that directory (shown here as testsuite):

  Create a configure.in file in this directory, to control configuration-dependent choices for your tests. So far as DejaGnu is concerned, the important thing is to set a value for the variable target_abbrev; this value is the link to the init file you will write soon. (For simplicity, we assume the environment is Unix, and use unix as the value.)

  What else is needed in configure.in depends on the requirements of your tool, your intended test environments, and which configure system you use. This example is a minimal configure.in for use with GNU Autoconf.

• Create Makefile.in (if you are using Autoconf), or Makefile.am(if you are using Automake), the source file used by configure to build your Makefile. If you are using GNU Automake.just add the keyword dejagnu to the AUTOMAKE_OPTIONS variable in your Makefile.am file. This will add all the Makefile support needed to run DejaGnu, and support the Make Check target.

  You also need to include two targets important to DejaGnu: check, to run the tests, and site.exp, to set up the Tcl copies of configuration-dependent values. This is called the Local Config File The check target must run the runtest program to execute the tests.

  The site.exp target should usually set up (among other things) the $tool variable for the name of your program. If the local site.exp file is setup correctly, it is possible to execute the tests by merely typing runtest on the command line.

  Example 35. Sample Makefile.in Fragment

  # Look for a local version of DejaGnu, otherwise use one in the path RUNTEST = `if test -f $(top_srcdir)/../dejagnu/runtest; then \ echo $(top_srcdir) ../dejagnu/runtest; \ else \ echo runtest; \ fi` # The flags to pass to runtest RUNTESTFLAGS = # Execute the tests check: site.exp all $(RUNTEST) $(RUNTESTFLAGS) \ --tool ${example} --srcdir $(srcdir) # Make the local config file site.exp: ./config.status Makefile @echo "Making a new config file..." -@rm -f ./tmp? @touch site.exp -@mv site.exp site.bak @echo "## these variables are automatically\ generated by make ##" > ./tmp0 @echo "# Do not edit here. If you wish to\ override these values" >> ./tmp0 @echo "# add them to the last section" >> ./tmp0 @echo "set host_os ${host_os}" >> ./tmp0 @echo "set host_alias ${host_alias}" >> ./tmp0 @echo "set host_cpu ${host_cpu}" >> ./tmp0 @echo "set host_vendor ${host_vendor}" >> ./tmp0 @echo "set target_os ${target_os}" >> ./tmp0 @echo "set target_alias ${target_alias}" >> ./tmp0 @echo "set target_cpu ${target_cpu}" >> ./tmp0 @echo "set target_vendor ${target_vendor}" >> ./tmp0 @echo "set host_triplet ${host_canonical}" >> ./tmp0 @echo "set target_triplet ${target_canonical}">>./tmp0 @echo "set tool binutils" >> ./tmp0 @echo "set srcdir ${srcdir}" >> ./tmp0 @echo "set objdir `pwd`" >> ./tmp0 @echo "set ${examplename} ${example}" >> ./tmp0 @echo "## All variables above are generated by\ configure. Do Not Edit ##" >> ./tmp0 @cat ./tmp0 > site.exp @sed < site.bak \ -e '1,/^## All variables above are.*##/ d' \ >> site.exp -@rm -f ./tmp?

• Create a directory (in testsuite) called config. Make a Tool Init File in this directory. Its name must start with the target_abbrev value, or be named default.exp so call it config/unix.exp for our Unix based example. This is the file that contains the target-dependent procedures. Fortunately, on Unix, most of them do not have to do very much in order for runtest to run.

  If the program being tested is not interactive, you can get away with this minimal unix.exp to begin with:

  Example 36. Simple Batch Program Tool Init File

  proc foo_exit {} {} proc foo_version {} {}

  If the program being tested is interactive, however, you might as well define a start routine and invoke it by using an init file like this:

  Example 37. Simple Interactive Program Tool Init File

  proc foo_exit {} {} proc foo_version {} {} proc foo_start {} { global ${examplename} spawn ${examplename} expect { -re "" {} } } # Start the program running we want to test foo_start

• Create a directory whose name begins with your tool's name, to contain tests. For example, if your tool's name is gcc, then the directories all need to start with "gcc.".

• Create a sample test file. Its name must end with .exp. You can use first-try.exp. To begin with, just write there a line of Tcl code to issue a message.

  Example 38. Testing A New Tool Config

  send_user "Testing: one, two...\n"

• Back in the testsuite (top level) directory, run configure. Typically you do this while in the build directory. You may have to specify more of a path, if a suitable configure is not available in your execution path.

• e now ready to triumphantly type make check or runtest. You should see something like this:

  Example 39. Example Test Case Run

  Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 === example tests === Running ./example.0/first-try.exp ... Testing: one, two... === example Summary ===

  There is no output in the summary, because so far the example does not call any of the procedures that establish a test outcome.

• Write some real tests. For an interactive tool, you should probably write a real exit routine in fairly short order. In any case, you should also write a real version routine soon.

Adding A New Target

DejaGnu has some additional requirements for target support, beyond the general-purpose provisions of configure. DejaGnu must actively communicate with the target, rather than simply generating or managing code for the target architecture. Therefore, each tool requires an initialization module for each target. For new targets, you must supply a few Tcl procedures to adapt DejaGnu to the target. This permits DejaGnu itself to remain target independent.

Usually the best way to write a new initialization module is to edit an existing initialization module; some trial and error will be required. If necessary, you can use the @samp{--debug} option to see what is really going on.

When you code an initialization module, be generous in printing information controlled by the verbose procedure.

For cross targets, most of the work is in getting the communications right. Communications code (for several situations involving IP networks or serial lines) is available in a DejaGnu library file.

If you suspect a communication problem, try running the connection interactively from Expect. (There are three ways of running Expect as an interactive interpreter. You can run Expect with no arguments, and control it completely interactively; or you can use expect -i together with other command-line options and arguments; or you can run the command interpreter from any Expect procedure. Use return to get back to the calling procedure (if any), or return -tcl to make the calling procedure itself return to its caller; use exit or end-of-file to leave Expect altogether.) Run the program whose name is recorded in $connectmode, with the arguments in $targetname, to establish a connection. You should at least be able to get a prompt from any target that is physically connected.

Adding A New Board

Adding a new board consists of creating a new board config file. Examples are in dejagnu/baseboards. Usually to make a new board file, it's easiest to copy an existing one. It is also possible to have your file be based on a baseboard file with only one or two changes needed. Typically, this can be as simple as just changing the linker script. Once the new baseboard file is done, add it to the boards_DATA list in the dejagnu/baseboards/Makefile.am, and regenerate the Makefile.in using automake. Then just rebuild and install DejaGnu. You can test it by:

There is a crude inheritance scheme going on with board files, so you can include one board file into another, The two main procedures used to do this are load_generic_config and load_base_board_description. The generic config file contains other procedures used for a certain class of target. The board description file is where the board specfic settings go. Commonly there are similar target environments with just different processors.

      make check RUNTESTFLAGS="--target_board=newboardfile".
      

Here's an example of a board config file. There are several helper procedures used in this example. A helper procedure is one that look for a tool of files in commonly installed locations. These are mostly used when testing in the build tree, because the executables to be tested are in the same tree as the new dejagnu files. The helper procedures are the ones in square braces [], which is the Tcl execution characters.

      # Load the generic configuration for this board. This will define a basic
      # set of routines needed by the tool to communicate with the board.
      load_generic_config "sim"

      # basic-sim.exp is a basic description for the standard Cygnus simulator.
      load_base_board_description "basic-sim"

      # The compiler used to build for this board. This has *nothing* to do
      # with what compiler is tested if we're testing gcc.
      set_board_info compiler "[find_gcc]"

      # We only support newlib on this target.
      # However, we include libgloss so we can find the linker scripts.
      set_board_info cflags "[newlib_include_flags] [libgloss_include_flags]"
      set_board_info ldflags "[newlib_link_flags]"

      # No linker script for this board.
      set_board_info ldscript "-Tsim.ld";

      # The simulator doesn't return exit statuses and we need to indicate this.
      set_board_info needs_status_wrapper 1

      # Can't pass arguments to this target.
      set_board_info noargs 1

      # No signals.
      set_board_info gdb,nosignals 1

      # And it can't call functions.
      set_board_info gdb,cannot_call_functions 1

      

Board Config File Values

These fields are all in the board_info These are all set by using the set_board_info procedure. The parameters are the field name, followed by the value to set the field to.

These fields are used by the GCC and GDB tests, and are mostly only useful to somewhat trying to debug a new board file for one of these tools. Many of these are used only by a few testcases, and their purpose is esoteric. These are listed with sample values as a guide to better guessing if you need to change any of these.

Writing A Test Case

The easiest way to prepare a new test case is to base it on an existing one for a similar situation. There are two major categories of tests: batch or interactive. Batch oriented tests are usually easier to write.

The GCC tests are a good example of batch oriented tests. All GCC tests consist primarily of a call to a single common procedure, Since all the tests either have no output, or only have a few warning messages when successfully compiled. Any non-warning output is a test failure. All the C code needed is kept in the test directory. The test driver, written in Tcl, need only get a listing of all the C files in the directory, and compile them all using a generic procedure. This procedure and a few others supporting for these tests are kept in the library module lib/c-torture.exp in the GCC test suite. Most tests of this kind use very few expect features, and are coded almost purely in Tcl.

Writing the complete suite of C tests, then, consisted of these steps:

• Copying all the C code into the test directory. These tests were based on the C-torture test created by Torbjorn Granlund (on behalf of the Free Software Foundation) for GCC development.

• Writing (and debugging) the generic Tcl procedures for compilation.

• Writing the simple test driver: its main task is to search the directory (using the Tcl procedure glob for filename expansion with wildcards) and call a Tcl procedure with each filename. It also checks for a few errors from the testing procedure.

Testing interactive programs is intrinsically more complex. Tests for most interactive programs require some trial and error before they are complete.

However, some interactive programs can be tested in a simple fashion reminiscent of batch tests. For example, prior to the creation of DejaGnu, the GDB distribution already included a wide-ranging testing procedure. This procedure was very robust, and had already undergone much more debugging and error checking than many recent DejaGnu test cases. Accordingly, the best approach was simply to encapsulate the existing GDB tests, for reporting purposes. Thereafter, new GDB tests built up a family of Tcl procedures specialized for GDB testing.

Debugging A Test Case

These are the kinds of debugging information available from DejaGnu:

• Output controlled by test scripts themselves, explicitly allowed for by the test author. This kind of debugging output appears in the detailed output recorded in the DejaGnu log file. To do the same for new tests, use the verbose procedure (which in turn uses the variable also called verbose) to control how much output to generate. This will make it easier for other people running the test to debug it if necessary. Whenever possible, if $verbose is 0, there should be no output other than the output from pass, fail, error, and warning. Then, to whatever extent is appropriate for the particular test, allow successively higher values of $verbose to generate more information. Be kind to other programmers who use your tests: provide for a lot of debugging information.

• Output from the internal debugging functions of Tcl and Expect. There is a command line options for each; both forms of debugging output are recorded in the file dbg.log in the current directory.

  Use --debug for information from the expect level; it generates displays of the expect attempts to match the tool output with the patterns specified. This output can be very helpful while developing test scripts, since it shows precisely the characters received. Iterating between the latest attempt at a new test script and the corresponding dbg.log can allow you to create the final patterns by ``cut and paste''. This is sometimes the best way to write a test case.

• Use --strace to see more detail at the Tcl level; this shows how Tcl procedure definitions expand, as they execute. The associated number controls the depth of definitions expanded.

• Finally, if the value of verbose is 3 or greater,DejaGnu turns on the expect command log_user. This command prints all expect actions to the expect standard output, to the detailed log file, and (if --debug is on) to dbg.log.

Adding A Test Case To A Testsuite.

There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The other is to create a new directory to hold your test. The existing test directories represent several styles of testing, all of which are slightly different; examine the directories for the tool of interest to see which (if any) is most suitable.

Adding a GCC test can be very simple: just add the C code to any directory beginning with gcc. and it runs on the next

runtest --tool
      gcc

To add a test to GDB, first add any source code you will need to the test directory. Then you can either create a new expect file, or add your test to an existing one (any file with a .exp suffix). Creating a new .exp file is probably a better idea if the test is significantly different from existing tests. Adding it as a separate file also makes upgrading easier. If the C code has to be already compiled before the test will run, then you'll have to add it to the Makefile.in file for that test directory, then run configure and make.

Adding a test by creating a new directory is very similar:

• Create the new directory. All subdirectory names begin with the name of the tool to test; e.g. G++ tests might be in a directory called g++.other. There can be multiple test directories that start with the same tool name (such as g++).

• Add the new directory name to the configdirs definition in the configure.in file for the testsuite directory. This way when make and configure next run, they include the new directory.

• Add the new test case to the directory, as above.

• To add support in the new directory for configure and make, you must also create a Makefile.in and a configure.in.

Hints On Writing A Test Case

It is safest to write patterns that match all the output generated by the tested program; this is called closure. If a pattern does not match the entire output, any output that remains will be examined by the next expect command. In this situation, the precise boundary that determines which expect command sees what is very sensitive to timing between the Expect task and the task running the tested tool. As a result, the test may sometimes appear to work, but is likely to have unpredictable results. (This problem is particularly likely for interactive tools, but can also affect batch tools---especially for tests that take a long time to finish.) The best way to ensure closure is to use the -re option for the expect command to write the pattern as a full regular expressions; then you can match the end of output using a $. It is also a good idea to write patterns that match all available output by using .*\ after the text of interest; this will also match any intervening blank lines. Sometimes an alternative is to match end of line using \r or \n, but this is usually too dependent on terminal settings.

Always escape punctuation, such as ( or ", in your patterns; for example, write \(. If you forget to escape punctuation, you will usually see an error message like

extra
      characters after close-quote.

If you have trouble understanding why a pattern does not match the program output, try using the --debug option to runtest, and examine the debug log carefully.

Be careful not to neglect output generated by setup rather than by the interesting parts of a test case. For example, while testing GDB, I issue a send set height 0\n command. The purpose is simply to make sure GDB never calls a paging program. The set height command in GDB does not generate any output; but running any command makes GDB issue a new (gdb) prompt. If there were no expect command to match this prompt, the output (gdb) begins the text seen by the next expect command---which might make that pattern fail to match.

To preserve basic sanity, I also recommended that no test ever pass if there was any kind of problem in the test case. To take an extreme case, tests that pass even when the tool will not spawn are misleading. Ideally, a test in this sort of situation should not fail either. Instead, print an error message by calling one of the DejaGnu procedures error or warning.

Special variables used by test cases.

There are special variables used by test cases. These contain other information from DejaGnu. Your test cases can use these variables, with conventional meanings (as well as the variables saved in site.exp. You can use the value of these variables, but they should never be changed.