Getting DejaGnu up and running

Windows

On Windows systems DejaGnu is part of a port of a lot of Unix tools to the Windows OS, called Cygwin. Cygwin may be downloaded and installed from a mirror of http://www.cygwin.com/. All examples were also run on Windows NT. If nothing is said, you can assume that you should get the same output as on a Unix system.

You will need a telnet daemon if you want to use a Windows box as a remote target. There seems to be a freeware telnet daemon at http://www.fictional.net/.

Getting the source code for the calc example

If you are running a Debian distribution you can find the examples under /usr/share/doc/dejagnu/examples. These examples seem to be missing in Red Hat's RPM. In this case download the sources of DejaGnu and adjust the pathes to the DejaGnu examples accordingly.

A simple project without the GNU autotools

The runtest program can be run standalone. All the autoconf/automake support is just cause those programs are commonly used for other GNU applications. The key to running runtest standalone is having the local site.exp file setup correctly, which automake does.

The generated site.exp should like like:

set tool calc
set srcdir .
set objdir /home/dgt/dejagnu.test

Using autoconf/autoheader/automake

We have to prepare some input file in order to run autocon and automake. There is book "GNU autoconf, automake and libtool" by Garry V. Vaughan, et al. NewRider, ISBN 1-57870-190-2 which describes this process thoroughly.

From the calc example distributed with the DejaGnu documentation you should copy the program file itself (calc.c) and some additional files, which you might examine a little bit close to derive their meanings.

dgt:~/dejagnu.test$ cp -r /usr/share/doc/dejagnu/examples/calc/\
{configure.in,Makefile.am,calc.c,testsuite} .

In Makemake.am note the presence of the AUTOMAKE_OPTIONS = dejagnu. This option is needed.

Run aclocal to generate aclocal.m4, which is a collection of macros needed by configure.in

dgt:~/dejagnu.test$ aclocal

autoconf is another part of the auto-tools. Run it to generate configure based on information contained in configure.in.

dgt:~/dejagnu.test$ autoconf

autoheader is another part of the auto-tools. Run it to generate calc.h.in.

dgt:~/dejagnu.test$ autoheader

The Makefile.am of this example was developed as port of the DejaGnu distribution. Adapt Makefile.am for this test. Replace the line "#noinst_PROGRAMS = calc" to "bin_PROGRAMS = calc". Change the RUNTESTDEFAULTFLAGS from "$$srcdir/testsuite" to "./testsuite".

Running automake at this point contains a series of warning in its output as shown in the following example:

dgt:~/dejagnu.test$ automake --add-missing
automake: configure.in: installing `./install-sh'
automake: configure.in: installing `./mkinstalldirs'
automake: configure.in: installing `./missing'
automake: Makefile.am: installing `./INSTALL'
automake: Makefile.am: required file `./NEWS' not found
automake: Makefile.am: required file `./README' not found
automake: Makefile.am: installing `./COPYING'
automake: Makefile.am: required file `./AUTHORS' not found
automake: Makefile.am: required file `./ChangeLog' not found
configure.in: 4: required file `./calc.h.in' not found
Makefile.am:6: required directory ./doc does not exist

Create a empty directory doc and empty files INSTALL, NEWS, README, AUTHORS, ChangeLog and COPYING. The default COPYING will point to the GNU Public License (GPL). In a real project it would be time to add some meaningfull text in each file.

Adapt calc to your environment by calling configure.

dgt:~/dejagnu.test$ ./configure
creating cache ./config.cache
checking whether to enable maintainer-specific portions of Makefiles... no
checking for a BSD compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking whether make sets ${MAKE}... yes
checking for working aclocal... found
checking for working autoconf... found
checking for working automake... found
checking for working autoheader... found
checking for working makeinfo... found
checking for gcc... gcc checking whether the C compiler (gcc ) works... yes
checking whether the C compiler (gcc ) is a cross-compiler... no
checking whether we are using GNU C... yes
checking whether gcc accepts -g... yes
checking for a BSD compatible install... /usr/bin/install -c
checking how to run the C preprocessor... gcc -E
checking for stdlib.h... yes
checking for strcmp... yes
updating cache ./config.cache
creating ./config.status
creating Makefile creating calc.h

If you are familiar with GNU software, this output should not contain any surprise to you. Any errors should be easy to fix for such a simple program.

Build the calc executable:

dgt:~/dejagnu.test$ make
gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c
gcc -g -O2 -o calc calc.o

You prepared a few files and then called some commands. Respecting the right order assures a automatic and correctly compiled calc program. The following example resumes the correct order.

dgt:~/dejagnu.test$ aclocal
dgt:~/dejagnu.test$ autoconf
dgt:~/dejagnu.test$ autoheader
dgt:~/dejagnu.test$ automake --add-missing
dgt:~/dejagnu.test$ ./configure
dgt:~/dejagnu.test$ make

Play with calc and verify whether it works correctly. A sample session might look like this:

dgt:~/dejagnu.test$ ./calc
calc: version
Version: 1.1
calc: add 3 4
7
calc: multiply 3 4 
12
calc: multiply 2 4 
12
calc: quit

Look at the intentional bug that 2 times 4 equals 12.

The tests run by DejaGnu need a file called site.exp, which is automatically generated if we call "make site.exp". This was the purpose of the "AUTOMAKE_OPTIONS = dejagnu" in Makefile.am.

dgt: make site.exp
dgt:~/dejagnu.test$ make site.exp
Making a new site.exp file...

Running the test for the calc example

Now we are ready to call the automated tests

dgt:~/dejagnu.test$ make check
make check-DEJAGNU
make[1]: Entering directory `/home/dgt/dejagnu.test' srcdir=`cd . && pwd`; export srcdir; \
EXPECT=expect; export EXPECT; \ runtest=runtest; \
if /bin/sh -c "$runtest --version" > /dev/null 2>&1; then \
$runtest --tool calc CALC=`pwd`/calc --srcdir ./testsuite ; \
else echo "WARNING: could not find \`runtest'" 1>&2; :;\
fi
WARNING: Couldn't find the global config file.
WARNING: Couldn't find tool init file
Test Run By dgt on Sun Nov 25 21:42:21 2001
Native configuration is i586-pc-linux-gnu

       === calc tests ===

Schedule of variations:
   unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./testsuite/config/unix.exp as tool-and-target-specific interface file.
Running ./testsuite/calc.test/calc.exp ...
FAIL: multiply2 (bad match)

=== calc Summary ===

# of expected passes 5
# of unexpected failures 1
/home/Dgt/dejagnu.test/calc version Version: 1.1
make[1]: *** [check-DEJAGNU] Fehler 1
make[1]: Leaving directory `/home/Dgt/dejagnu.test' make: *** [check-am] Fehler 2

Did you see the line "FAIL:"? The test cases for calc catch the bug in the calc.c file. Fix the error in calc.c later as the following examples assume a unchanged calc.c.

Examine the output files calc.sum and calc.log. Try to understand the testcases written in ~/dejagnu.test/testsuite/calc.test/calc.exp. To understand Expect you might take a look at the book "Exploring Expect", which is an excellent resource for learning and using Expect. (Pub: O'Reilly, ISBN 1-56592-090-2) The book contains hundreds of examples and also includes a tutorial on Tcl. Exploring Expect is 602 pages long.

The various config files or how to avoid warnings

DejaGnu may be customized by each user. It first searches for a file called ~/.dejagnurc. Create the file ~/.dejagnurc and insert the following line:

puts "I am ~/.dejagnurc"

Rerun make check. Test whether the output contains "I am ~/.dejagnurc". Create ~/my_dejagnu.exp and insert the following line:

puts "I am ~/my_dejagnu.exp"

In a Bash-Shell enter

dgt:~/dejagnu.test$ export DEJAGNU=~/my_dejagnu.exp

Run "make check" again. The output should not contain "WARNING: Couldn't find the global config file.". Create the sub-directory lib. Create the file "calc.exp" in it and insert the following line:

puts "I am lib/calc.exp"

The last warning "WARNING: Couldn't find tool init file" should not be part of the output of make check. Create the directory ~/boards. Create the file ~/boards/standard.exp and insert the following line:

puts "I am boards/standard.exp"

If the variable DEJAGNU is still not empty then the (abbreviated) output of "make check" should look like this:

dgt:~/dejagnu.test$ make check
<...>
fi
I am ~/.dejagnurc
I am ~/my_dejagnu.exp
I am lib/calc.exp
Test Run By dgt on Sun Nov 25 22:19:14 2001
Native configuration is i586-pc-linux-gnu

     === calc tests ===
Using /home/Dgt/boards/standard.exp as standard board description\
file for build.
I am ~/boards/standard.exp
Using /home/Dgt/boards/standard.exp as standard board description\
 file for host.
I am ~/boards/standard.exp

Schedule of variations:
  unix

Running target unix
Using /home/Dgt/boards/standard.exp as standard board description\
 file for target.
I am ~/boards/standard.exp
Using /usr/share/dejagnu/baseboards/unix.exp as board description file\
for target.
<...>

It is up to you to decide when and where to use any of the above mentioned config files for customizing. This chapters showed you where and in which order the different config files are run.

When trouble strikes

Calling runtest with the '-v'-flag shows you in even more details which files are searched in which order. Passing it several times gives more and more details.

runtest -v -v -v --tool calc CALC=`pwd`/calc --srcdir ./testsuite

Calling runtest with the '--debug'-flag logs a lot of details to dbg.log where you can analyse it afterwards.

In all test cases you can temporary adjust the verbosity of information by adding the following Tcl-command to any tcl file that gets loaded by dejagnu, for instance, ~/.dejagnurc:

set verbose 9

Testing "Hello world" locally

This test checks, whether the built-in shell command "echo Hello world" will really write "Hello world" on the console. Create the file ~/dejagnu.test/testsuite/calc.test/local_echo.exp. It should contain the following lines

set test "Local Hello World"
send "echo Hello World"
expect {
   -re "Hello World"  { pass "$test" }
}

Run runtest again and verify the output "calc.log"

Setup telnet to your own host

The easiest remote host is usually the host you are working on. In this example we will use telnet to login in your own workstation. For security reason you should never have a telnet deamon running on machine connected on the internet, as password and usernames are transmitted in clear text. We assume you know how to setup your machine for a telnet daemon.

Next try whether you may login in your own host by issuing the command "telnet localhost.1". In order to be able to distinguish between a normal session an a telnet login add the following lines to /home/dgt/.bashrc.

if [ "$REMOTEHOST" ]
then
   PS1='remote:\w\$ '
fi

Now on the machine a "remote" login looks like this:

dgt:~/dejagnu.test$ telnet localhost
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Debian GNU/Linux testing/unstable Linux
K6Linux login: dgt
Password:
Last login: Sun Nov 25 22:46:34 2001 from localhost on pts/4
Linux K6Linux 2.4.14 #1 Fre Nov 16 19:28:25 CET 2001 i586 unknown
No mail.
remote:~$ exit
logout
Connection closed by foreign host.

A test case for login via telnet

In order to define a correct setup we have add a line containing "set target unix" either to ~/.dejagnurc or to ~/my_dejagnu.exp. In ~/boards/standard.exp add the following four lines to define a few patterns for the DejaGnu telnet login procedure.

set_board_info shell_prompt    "remote:"
set_board_info telnet_username "dgt"
set_board_info telnet_password "top_secret"
set_board_info hostname        "localhost"

As DejaGnu will be parsing the telnet session output for some well known pattern the output there are a lot of things that can go wrong. If you have any problems verify your setup:

• Is /etc/motd empty?

• Is /etc/issue.net empty?

• Exists a empty ~/.hushlogin?

• The LANG environment variable must be either empty or set to "C".

To test the login via telnet write a sample test case. Create the file ~/dejagnu.test/testsuite/calc.test/remote_echo.exp and add the following few lines:

puts "this is remote_echo.exp target for $target "
target_info $target
#set verbose 9
set shell_id [remote_open $target]
set test "Remote login to $target"
#set verbose 0
puts "Spawn id for remote shell is $shell_id"
if { $shell_id > 0 } {
   pass "$test"
} else {
   fail "Remote open to $target"
}

In the runtest output you should find something like:

Running ./testsuite/calc.test/local_echo.exp ...
Running ./testsuite/calc.test/remote_echoo.exp ...
this is remote_echo.exp target is unix
Spawn id for remote shell is exp7

Have again a look at calc.log to get a feeling how DejaGnu and expect parse the input.

Remote testing "Hello world"

Next you will transform the above "hello world" example to its remote equivalent. This can be done by adding the following lines to our file remote_echo.exp.

set test "Remote_send Hello World"
set status [remote_send $target "echo \"Hello\" \"World\"\n" ]
pass "$test"
set test "Remote_expect Hello World"
remote_expect $target 5 {
   -re "Hello World"  { pass "$test" }
}

Call make check. The output should contain "# of expected passes 9" and "# of unexcpected failures 1".

Have a look at the procedures in /usr/share/dejagnu/remote.exp to have an overview of the offered procedures and their features.

Now setup a real target. In the following example we assume as target a PowerBook running Debian. As above add a test user "dgt", install telnet and FTP servers. In order to distinguish it from the host add the line

PS1='test:>'

In order to let runtest run its test on the "powerbook" target change the following lines in ~/boards/standard.exp:

set_board_info protocol        "telnet"
set_board_info telnet_username "dgt"
set_board_info telnet_password "top_secret"
set_board_info shell_prompt    "test:> "
set_board_info hostname        "powerbook"

Now call runtest again with the same arguments and verify whether all went okay by taking a close look at calc.log.

Transferring files from/to the target

A simple procedure like this will do the job for you:

set test "Remote_download"
puts "Running Remote_download"
# set verbose 9
set remfile /home/dgt/dejagnu2

set status [remote_download $target /home/dgt/.dejagnurc $remfile]
if { "$status" == "" } {
     fail "Remote download to $remfile on $target"
} else {
   pass "$test"
}

puts "status of remote_download ist $status"
# set verbose 0

After running runtest again, check whether the file dejagnu2 exists on the target. This example will only work if the rcp command works with your target. If you have a working FTP-server on the target you can use it by adding the following lines to ~/boards/standard.exp:

set_board_info file_transfer   "ftp"
set_board_info ftp_username    "dgt"
set_board_info ftp_password    "1234"

Preparing for crosscompilation

For crosscompiling you need working binutils, gcc and a base library like libc or glib for your target. It is beyond the scope of this document to describe how to get it working. The following examples assume a cross compiler for PowerPC which is called linux-powerpc-gcc.

Add AC_CANONICAL_TARGET in dejagnu.test/configure.in at the following location. Copy config.guess from /usr/share/automake to dejagnu.test.

AM_CONFIG_HEADER(calc.h)
AC_CANONICAL_TARGET([])
AM_INIT_AUTOMAKE(calc, 1.1)

You need to run automake 2.5 or later. Depending on your installation calling autoconf2.5 instead of autoconf is not needed. The sequence to regenerate all files is:

$ autoconf2.5
$ autoheader
$ automake
$ ./configure --host=powerpc-linux --target=powerpc-linux
configure: WARNING: If you wanted to set the --build type, don't use --host.
    If a cross compiler is detected then cross compile mode will be used.
checking build system type... ./config.guess: ./config.guess: No such file or directory
configure: error: cannot guess build type; you must specify one
$ cp /usr/share/automake/config.guess .
$ ./configure --host=powerpc-linux --target=powerpc-linux
configure: WARNING: If you wanted to set the --build type, don't use --host.
If a cross compiler is detected then cross compile mode will be used. \
checking build system type... i586-pc-linux-gnu
checking host system type... powerpc-unknown-linux-gnu
<...>
checking whether we are cross compiling... yes
<...>
Configuration:
Source code location: .
C Compiler: powerpc-linux-gcc
C Compiler flags: -g -O2

Everything should be ready to recompile for the target:

$ make
powerpc-linux-gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c
powerpc-linux-gcc -g -O2 -o calc calc.o

Remote testing of calc

Not yet written, as I have problem getting libc6-dev-powerpc to work. Probably I first have to build my cross compiler.

Using Windows as host and vxWorks as target

A more thorough walk-through will be written in a few weeks.

In order to test the vxWorks as a target I changed boards/standards.exp to reflect my settings (IP, username, password). Then I reconfigured vxWorks to include a FTP and telnet server (using the same username/password combination ad in boards/standard.exp).

With this setup and some minor modification (e.g. replacing echo by printf) in my test cases I could test my vxWorks system. It sure does not seem to be a correct setup by DejaGnu standard. For instance, it still loading /usr/share/dejagnu/baseboards/unix.exp instead of vxWorks. In any case I found that (at least under Windows) I did not find out how the command line would let me override settings in my personal config files.