Overview

What is DejaGnu ?

DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl -- Tool command language. There is more information on Tcl at the Scriptics web site and the Expect web site is at NIST.

Julia Menapace first coined the term ``DejaGnu'' to describe an earlier testing framework at Cygnus Support she had written for GDB. When we replaced it with the Expect-based framework, it was like DejaGnu all over again. More importantly, it was also named after my daughter, Deja Snow Savoye (now 14 years old as of Feb 2004), who was a toddler during DejaGnu's beginnings.

DejaGnu offers several advantages for testing:

• The flexibility and consistency of the DejaGnu framework make it easy to write tests for any program, with either batch oriented, or interactive programs.

• DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run from any supported host system on any supported target system. DejaGnu runs tests on many single board computers, whose operating software ranges from a simple boot monitor to a real-time OS.

• All tests have the same output format. This makes it easy to integrate testing into other software development processes. DejaGnu's output is designed to be parsed by other filtering script and it is also human readable.

• Using Tcl and Expect, it's easy to create wrappers for existing testsuites. By incorporating existing tests under DejaGnu, it's easier to have a single set of report analyse programs..

Running tests requires two things: the testing framework and the testsuites themselves. Tests are usually written in Expect using Tcl, but you can also use a Tcl script to run a testsuite that is not based on Expect. Expect script filenames conventionally use .exp as a suffix; for example, the main implementation of the DejaGnu test driver is in the file runtest.exp.)

What's New In This Release

This release has a number of substantial changes over version 1.3. The most visible change is that the version of Expect and Tcl included in the release are up-to-date with the current stable net releases. The biggest change is years of modifications to the target configuration system, used for cross testing. While this greatly improved cross testing, is has made that subsystem very complicated. The goal is to have this entirely rewritten using iTcl by the next release. Other changes are:

• More built-in support for building target binaries with the correct linker flags. Currently this only works with GCC as the cross compiler, preferably with a target supported by Libgloss.

• Lots of little bug fixes from years of heavy use at Cygnus Solutions.

• DejaGnu now uses Automake for Makefile configuration.

• Updated documentation, now in DocBook XML.

• Windows support. There is beta level support for Windows that is still a work in progress. This requires the Cygwin POSIX subsystem for Windows.

Design Goals

DejaGnu grew out of the internal needs of Cygnus Solutions, the company formerly known as Cygnus Support. Cygnus maintained and enhanced a variety of free programs in many different environments and we needed a testing tool that:

• was useful to developers while fixing bugs;

• automated running many tests during a software release process;

• was portable among a variety of host computers;

• supported cross-development testing;

• permitted testing interactive programs, like GDB; and

• permitted testing batch oriented programs, like GCC.

Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration.

Probably the greatest challenge was testing in a cross-development environment. Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to Ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required and supported thereafter. Once a communication procedure is coded, any test can use it. Currently DejaGnu can use rsh, rlogin, telnet, tip, kermit and mondfe for remote communications.

A POSIX conforming test framework

DejaGnu conforms to the POSIX 1003.3 standard for test frameworks. Rob Savoye was a member of that committee.

The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of X/Open, Unix International and the Open Software Foundation.

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no testsuite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of POSIX conforming systems and a fifth for a conditional feature. DejaGnu supports the use of all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

• A test's execution is interrupted.

• A test does not produce a clear result. This is usually because there was an ERROR from DejaGnu while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened---and to improve the test case.

• A test depends on a previous test, which fails.

• The test was set up incorrectly.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

DejaGnu uses the same output procedures to produce these messages for all testsuites and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, you must avoid the setupxfail} procedure as described in the PASS section above and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above.