.\" Motif
.\"
.\" Copyright (c) 1987-2012, The Open Group. All rights reserved.
.\"
.\" These libraries and programs are free software; you can
.\" redistribute them and/or modify them under the terms of the GNU
.\" Lesser General Public License as published by the Free Software
.\" Foundation; either version 2 of the License, or (at your option)
.\" any later version.
.\"
.\" These libraries and programs are distributed in the hope that
.\" they will be useful, but WITHOUT ANY WARRANTY; without even the
.\" implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
.\" PURPOSE. See the GNU Lesser General Public License for more
.\" details.
.\"
.\" You should have received a copy of the GNU Lesser General Public
.\" License along with these librararies and programs; if not, write
.\" to the Free Software Foundation, Inc., 51 Franklin Street, Fifth
.\" Floor, Boston, MA 02110-1301 USA
...\" 
...\" 
...\" HISTORY
# $XConsortium: ch04.sm /main/3 1995/07/13 20:09:16 drk $
...\"  (c) Copyright 1992 OPEN SOFTWARE FOUNDATION, INC.
.H 1 "Debugging the Automated Tests"
.P
Interpreting test results often requires debugging each individual
discrepancy until its cause can be determined. The follow sections
describe how the automation technology supports debugging.
.H 2 "Types of Failures"
.P
This section describes the types of failures which may be found by automated
testing. The next section provides some hints on how to go about debugging
those failures.
.H 3 "Visual Comparison Failures"
.P
A common visual comparison failure occurs when the original, recorded image
differs in size from the current image under test. Because the automation
code does not know how to compare different-sized images, it simply issues
a warning. In interactive mode, the original region will be
displayed in the dialog box, although the overlay feature is disabled.
One can, however, use the X client \*Lxmag\*O and count pixels to help
determine what has changed.
.P
When two images are the same size, the automation code does a pixel-by-pixel comparison of the images. If any one pixel differs between the
two images, the automation code prints out a message that a CompareVisual
failure has occurred.
In interactive mode, the tester can examine both
the recorded and current images, and overlay the differences (highlighted
in red) on top of either image.
.P
There are several problems that can produce a \*LVisual Image Not Found\*O
error:
the recorded
image is missing or corrupted; the environment variable
\*L$AUTOVPATH\*O is not defined; the directory defined by
\*L$AUTOVPATH\*O does not
exist, or the visual file in that directory is not readable or is missing.
.P
This error can also occur if the test or the script was modified between
the recording and the playback. For instance, a test is recorded
with a script that specifies six visual comparisons. Six images are
stored in the \*L.vis\*O file. Later, a tester adds an additional
\*LCompareVisual\*O command to the script but fails to rerecord the test.
On the next playback, the seventh visual comparison fails because no
image is stored with which to compare the current image.
.P
Either a \*LChecksum\*O or a \*LVisual Image Corrupt\*O error can occur
when something in the visuals file has become corrupted. The only way to
fix this problem is to rerecord the original, using the original baseline.
.H 3 "Fatal Errors"
.P
The signals are trapped by the automation code, and redirected to
a special routine \fBAutoExitSignal\*O which simply prints out a "signal received" message, and the type of signal. Signals so captured are:
.sp
.in +3
.VL 1i
.LI "\*LSIGHUP\*O"
hangup
.LI "\*LSIGINT\*O"
interrupt
.LI "\*LSIGQUIT\*O"
quit
.LI "\*LSIGKILL\*O"
kill
.LI "\*LSIGSEGV\*O"
segmentation violation
.LI "\*LSIGFPE\*O"
arithmetic (floating point) exception
.LI "\*LSIGILL\*O"
illegal instruction
.LI "\*LSIGTERM\*O"
software termination signal
.LI "\*LSIGABRT\*O"
abort
.LI "\*LSIGBUS\*O"
bus error
.LI "\*LSIGSYS\*O"
bad argument to system call
.LE
.in -3
.sp
.fi
.P
Be certain in reviewing the differences between files that you search for
signal messages.
.H 3 "Other Failures"
.P
Many other kinds of failures only show up by examining the contents of
the differences file (\*L.err\*O) produced by the RUN script; this differences file is the product of the command \*Ldiff\*O run against the output file stored at record time and the output file for the current run. This section
describes some examples of what differences might turn up; very subtle bugs
may make their appearance known only through small differences in the output.
.P
.BL
.LI
Messages printed by callback routines can indicate more or fewer callbacks are occurring on some action.
.LI
Differences in size reported by callback routines may show layout differences
not captured by CompareVisual, usually because the top-level manager is not photographed.
.LI
User data printed out may indicate a test was run in the wrong mode.
.LI
CompareVisual failure messages will turn up in diffs.
.LI
Running a non-automated \*Lmwm\*O can produce window command failures which show up as differences. This may precipitate CompareVisual failures further along in the test.
.LI
Key-binding error messages may appear in diffs if the \fBqauser\*O environment
in not used.
.LI
\*LxisMovePointer\*O errors may mean that layout has changed, and the pointer can
no longer move to a certain widget.
.LI
If tests are run in interactive mode, "Accept" messages will appear in diffs
but can be ignored.
.LE
.H 2 "Debugging Tricks and Testing Strategies"
.P
This section will describe several automation features which can be used
to help debug test failures. While nothing is as good as first-hand
experience, this section will give you some idea of the tools at your
disposal.
.P
Generally, one will wish to run the entire test suite automatically then
debug the failures one at a time. If the failures appear to be due (in whole
or in part) to visual differences, it often makes the most sense to run
the offending test in interactive mode. The various overlay capabilities of
interactive mode have been described earlier. This is the quickest way to
assess the nature of the change; once you know what the difference is you
can debug the cause. However, there are other modes and features which
can be used.
.H 3 "The \*L-a\*O Flag"
The precursor to the interactive window was the \*L-a\*O flag. When run
with this flag, tests print out the x,y location of each pixel that differs
from the original. If the differences are really a single or two pixels and
you have difficulty seeing the differences in interactive mode, the information provided by \*L-a\*O may be helpful.
.H 3 "Window Dumps"
Another visual debugging tool which predates the interactive mode window
is the window dump feature. Tests run with the
\*L-W\*O flag create xwd-format window dumps which can subsequently be
viewed with the \*Lxwud\*O client. Your company may have other tools that
use xwd-format images; if so, this option may be useful for you.
.H 3 "The \*LManual\*O Command"
For certain problems, you will need to be able to manipulate the widgets in
the test client directly. Because automation takes control of the pointer,
we have provided a scripting language command \*LManual\*O which tells
the automation code to suspend its normal event handling and return control
to the user, until the user clicks the Continue button on the
Manual mode DialogBox. At that point the
test will resume. This gives you an opportunity to examine closely what 
happens at very specific points during the test.
.P
To use \*LManual\*O, simply add the command at the appropriate point in
the test's \*L.scr\*O file and remake the \*L.Scr\*O file. When the
Manual command is reached, a special dialog box will appear and you may
exercise the test client normally. At this point, you can also invoke
other useful X debugging clients, such as \*Lxmag\*O, \*Lxwininfo\*O, 
or the OSF/Motif test tool \*Lxmruler\*O.
.H 3 "Delay Mode"
If you need the playback slowed enough for a human tester to watch the
changes occurring, but don't actually require the suspension of the event
loop provided by \*LManual\*O, the \*L-D\*O or \*LDelay\*O mode
might be helpful. Run with \*L-D \*Vseconds\*O, the code "sleeps" for the
requested number of seconds before executing the next instruction.
The Delay flag can also be used when very fast workstations execute 
instructions before the visuals have "caught up". 
.H 3 "Trace Mode"
The \*L-T\*O or \*LTrace\*O mode is useful in many debugging situations.
In Trace mode, the scripting instructions are echoed to the output file
as they are executed. This greatly simplies, for instance, the process of
determining which button produces an extra Activate callback. It may be
worth considering using the Trace mode by default for all recording and
playback. This can also help determine cases where failures are merely the
result of recorded output files getting out-of-sync with the tests or scripts
currently executing.
.H 3 "The \*Lxmruler\*O Tool"
The tool \*Lxmruler\*O is provided in the directory \*L$TOP/tests/tools\*O.
This client allows the user to "measure" widgets on screen which claim to
be a certain size.