.\" Motif

.\"

.\" Copyright (c) 1987-2012, The Open Group. All rights reserved.

.\"

.\" These libraries and programs are free software; you can

.\" redistribute them and/or modify them under the terms of the GNU

.\" Lesser General Public License as published by the Free Software

.\" Foundation; either version 2 of the License, or (at your option)

.\" any later version.

.\"

.\" These libraries and programs are distributed in the hope that

.\" they will be useful, but WITHOUT ANY WARRANTY; without even the

.\" implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR

.\" PURPOSE. See the GNU Lesser General Public License for more

.\" details.

.\"

.\" You should have received a copy of the GNU Lesser General Public

.\" License along with these librararies and programs; if not, write

.\" to the Free Software Foundation, Inc., 51 Franklin Street, Fifth

.\" Floor, Boston, MA 02110-1301 USA

...\" 

...\" 

...\" HISTORY

# $XConsortium: ch04.sm /main/3 1995/07/13 20:09:16 drk $

...\"  (c) Copyright 1992 OPEN SOFTWARE FOUNDATION, INC.

.H 1 "Debugging the Automated Tests"

.P

Interpreting test results often requires debugging each individual

discrepancy until its cause can be determined. The follow sections

describe how the automation technology supports debugging.

.H 2 "Types of Failures"

.P

This section describes the types of failures which may be found by automated

testing. The next section provides some hints on how to go about debugging

those failures.

.H 3 "Visual Comparison Failures"

.P

A common visual comparison failure occurs when the original, recorded image

differs in size from the current image under test. Because the automation

code does not know how to compare different-sized images, it simply issues

a warning. In interactive mode, the original region will be

displayed in the dialog box, although the overlay feature is disabled.

One can, however, use the X client \*Lxmag\*O and count pixels to help

determine what has changed.

.P

When two images are the same size, the automation code does a pixel-by-pixel comparison of the images. If any one pixel differs between the

two images, the automation code prints out a message that a CompareVisual

failure has occurred.

In interactive mode, the tester can examine both

the recorded and current images, and overlay the differences (highlighted

in red) on top of either image.

.P

There are several problems that can produce a \*LVisual Image Not Found\*O

error:

the recorded

image is missing or corrupted; the environment variable

\*L$AUTOVPATH\*O is not defined; the directory defined by

\*L$AUTOVPATH\*O does not

exist, or the visual file in that directory is not readable or is missing.

.P

This error can also occur if the test or the script was modified between

the recording and the playback. For instance, a test is recorded

with a script that specifies six visual comparisons. Six images are

stored in the \*L.vis\*O file. Later, a tester adds an additional

\*LCompareVisual\*O command to the script but fails to rerecord the test.

On the next playback, the seventh visual comparison fails because no

image is stored with which to compare the current image.

.P

Either a \*LChecksum\*O or a \*LVisual Image Corrupt\*O error can occur

when something in the visuals file has become corrupted. The only way to

fix this problem is to rerecord the original, using the original baseline.

.H 3 "Fatal Errors"

.P

The signals are trapped by the automation code, and redirected to

a special routine \fBAutoExitSignal\*O which simply prints out a "signal received" message, and the type of signal. Signals so captured are:

.sp

.in +3

.VL 1i

.LI "\*LSIGHUP\*O"

hangup

.LI "\*LSIGINT\*O"

interrupt

.LI "\*LSIGQUIT\*O"

quit

.LI "\*LSIGKILL\*O"

kill

.LI "\*LSIGSEGV\*O"

segmentation violation

.LI "\*LSIGFPE\*O"

arithmetic (floating point) exception

.LI "\*LSIGILL\*O"

illegal instruction

.LI "\*LSIGTERM\*O"

software termination signal

.LI "\*LSIGABRT\*O"

abort

.LI "\*LSIGBUS\*O"

bus error

.LI "\*LSIGSYS\*O"

bad argument to system call

.LE

.in -3

.sp

.fi

.P

Be certain in reviewing the differences between files that you search for

signal messages.

.H 3 "Other Failures"

.P

Many other kinds of failures only show up by examining the contents of

the differences file (\*L.err\*O) produced by the RUN script; this differences file is the product of the command \*Ldiff\*O run against the output file stored at record time and the output file for the current run. This section

describes some examples of what differences might turn up; very subtle bugs

may make their appearance known only through small differences in the output.

.P

.BL

.LI

Messages printed by callback routines can indicate more or fewer callbacks are occurring on some action.

.LI

Differences in size reported by callback routines may show layout differences

not captured by CompareVisual, usually because the top-level manager is not photographed.

.LI

User data printed out may indicate a test was run in the wrong mode.

.LI

CompareVisual failure messages will turn up in diffs.

.LI

Running a non-automated \*Lmwm\*O can produce window command failures which show up as differences. This may precipitate CompareVisual failures further along in the test.

.LI

Key-binding error messages may appear in diffs if the \fBqauser\*O environment

in not used.

.LI

\*LxisMovePointer\*O errors may mean that layout has changed, and the pointer can

no longer move to a certain widget.

.LI

If tests are run in interactive mode, "Accept" messages will appear in diffs

but can be ignored.

.LE

.H 2 "Debugging Tricks and Testing Strategies"

.P

This section will describe several automation features which can be used

to help debug test failures. While nothing is as good as first-hand

experience, this section will give you some idea of the tools at your

disposal.

.P

Generally, one will wish to run the entire test suite automatically then

debug the failures one at a time. If the failures appear to be due (in whole

or in part) to visual differences, it often makes the most sense to run

the offending test in interactive mode. The various overlay capabilities of

interactive mode have been described earlier. This is the quickest way to

assess the nature of the change; once you know what the difference is you

can debug the cause. However, there are other modes and features which

can be used.

.H 3 "The \*L-a\*O Flag"

The precursor to the interactive window was the \*L-a\*O flag. When run

with this flag, tests print out the x,y location of each pixel that differs

from the original. If the differences are really a single or two pixels and

you have difficulty seeing the differences in interactive mode, the information provided by \*L-a\*O may be helpful.

.H 3 "Window Dumps"

Another visual debugging tool which predates the interactive mode window

is the window dump feature. Tests run with the

\*L-W\*O flag create xwd-format window dumps which can subsequently be

viewed with the \*Lxwud\*O client. Your company may have other tools that

use xwd-format images; if so, this option may be useful for you.

.H 3 "The \*LManual\*O Command"

For certain problems, you will need to be able to manipulate the widgets in

the test client directly. Because automation takes control of the pointer,

we have provided a scripting language command \*LManual\*O which tells

the automation code to suspend its normal event handling and return control

to the user, until the user clicks the Continue button on the

Manual mode DialogBox. At that point the

test will resume. This gives you an opportunity to examine closely what 

happens at very specific points during the test.

.P

To use \*LManual\*O, simply add the command at the appropriate point in

the test's \*L.scr\*O file and remake the \*L.Scr\*O file. When the

Manual command is reached, a special dialog box will appear and you may

exercise the test client normally. At this point, you can also invoke

other useful X debugging clients, such as \*Lxmag\*O, \*Lxwininfo\*O, 

or the OSF/Motif test tool \*Lxmruler\*O.

.H 3 "Delay Mode"

If you need the playback slowed enough for a human tester to watch the

changes occurring, but don't actually require the suspension of the event

loop provided by \*LManual\*O, the \*L-D\*O or \*LDelay\*O mode

might be helpful. Run with \*L-D \*Vseconds\*O, the code "sleeps" for the

requested number of seconds before executing the next instruction.

The Delay flag can also be used when very fast workstations execute 

instructions before the visuals have "caught up". 

.H 3 "Trace Mode"

The \*L-T\*O or \*LTrace\*O mode is useful in many debugging situations.

In Trace mode, the scripting instructions are echoed to the output file

as they are executed. This greatly simplies, for instance, the process of

determining which button produces an extra Activate callback. It may be

worth considering using the Trace mode by default for all recording and

playback. This can also help determine cases where failures are merely the

result of recorded output files getting out-of-sync with the tests or scripts

currently executing.

.H 3 "The \*Lxmruler\*O Tool"

The tool \*Lxmruler\*O is provided in the directory \*L$TOP/tests/tools\*O.

This client allows the user to "measure" widgets on screen which claim to

be a certain size.