MPICH Test Suite

This test suite is a *supplement* to other test suites, including the
original MPICH testsuite, the Intel testsuite, and the IBM MPI test suite
(or test suites derived from that test, including the MPI C++ tests).

Building the Test Suite
=======================
In many cases, configure will find the MPI implementation
automatically.  In some cases, it will need some help.  For example:

For IBM MPI, where the compilation commands are not mpicc and mpif77 etc.:

./configure CC=xlc MPICC=mpcc F77=xlf MPIF77=mpxlf CXX=xlC \
                       MPICXX="mpCC -cpp" FC=xlf90 MPIFC=mpxlf90 \
		       --disable-spawn \
		       --enable-strictmpi

(or the _r versions of the compilers)

If mpicc and friends are not in your default path (and you do not want to
add them), you can specify the path with --with-mpi=<path>.  For example,
if they are in /usr/local/mympi/bin, use

./configure --with-mpi=/usr/local/mympi 

(configure will append the bin to the path that you give).

You may need to add MPI_SIZEOF_OFFSET=8 .

The option "-cpp" is needed for at least some versions of mpCC to define the
C++ bindings of the MPI routines.

For implementations that do not implement all of MPI-2, there are --disable
options, including --disable-spawn and --disable-cxx.  To restrict tests to 
just what is defined in the MPI specification, use --enable-strictmpi .

The script that runs the tests assumes that the MPI implementation
supports mpiexec; you should consider this the first test of the implementation.

Setting Options
===============
The following environment variables will modify the behavior of the tests

MPITEST_DEBUG - if set, output information for debugging the test suite
MPITEST_VERBOSE - if set to an integer value, output messages whose
                  level is at least that value (0 is a good choice here)
MPITEST_RETURN_WITH_CODE - Set the return code from the test programs based on
			 success or failure, with a zero for success and one
			 for failure (value must be yes, YES, true, or TRUE to 
			 turn this on)
MPITEST_THREADLEVEL_DEFAULT - Set the default thread level.  Values are 
			      multiple, serialized, funneled, and single.

Batch Systems
=============
For systems that run applications through a batch system, the option "-batch"
to the runtests script will create a script file that can be edited and 
submitted to the batch system.  The script checktests can be run to 
summarize the results.  

Specifically, (assuming the bash shell, and that the directory "btest", a
subdirectory of the test suite directory, is used for running the tests):

export MPITEST_BATCHDIR=`pwd`/btest
runtests -batch -tests=testlist
... edit btest/runtests.batch to make it a value batch submissions script
... run that script and wait for the batch job to complete
cd btest && ../checktests 

If a program other than mpiexec is used in the batch form to run programs, then
specify that to runtests:

    runtests -batch -mpiexec=aprun -tests=testlist

(Here, aprun is the command used on Cray XE6 systems.)

Note that some programs that are used to run MPI programs add extra output, 
which can confuse any tool that depends on clean output in STDOUT.  Since
such unfortunate behavior is common, the option -ignorebogus can be given 
to checktests:

cd btest && ../checktests --ignorebogus

See "More control over running tests" to see how to control how many
processes per node on used.  For example, on a Cray XE-6, this command
line to runtests can be used:

  runtests -batch -tests=testlist -ppnarg="-N %d" -ppn=2 -showprogress \
           -mpiexec=aprun

This runs at most 2 processes per node.  Note that this can take a long
time to execute because it builds all of the executables required for the
tests (over 800 of them!).  The "-showprogress" flag lets you know that
something is happening, but is not necessary.

Controlling the Tests that are Run
==================================
The tests are actually built and run by the script "runtests".  This script
can be given a file that contains a list of the tests to run.  This file has
two primary types of entries:

    directories:  Enter directory and look for the file "testlist".
                  Recursively run the contents of that file
    program names: Build and run that program

Lines may also be commented out with "#".

The simplest program line contains the name of the program and the number of
MPI processes to use.  For example, the following will build the
program sendrecv1 and run it with 4 processes:

sendrecv1 4

In addition, the program line can contain key=value pairs that provide 
special information about running the test.  For example, 

sendflood 8 timeLimit=600

says to build and run the program sendflood with 8 MPI processes and
permit the test to run for 600 seconds (by default, at least for
MPICH, the default timelimit is 180 seconds).  Other key=value pairs
can be used to select whether a program should be run at all,
depending on the abilities of the MPI implementation (this is
particularly important for Fortran programs, since preprocessor
support for Fortran is a non-standard extension to the Fortran
language, and there are some compilers that would not accept Fortran
programs that used the preprocessor).

The most important key=value pairs are:


timeLimit=n : Use a timelimit of n seconds

arg=string  : Run the program with string as an argument to the program

mpiexecarg=string  : Run the program with string as an argument to mpiexec

env=name=value : Run the program with environment variable "name" given the
                 value "value"

mpiversion=x.y : Build and run the program only if the MPI version is at 
                 least x.y.  For example, 

                distgraph1 4 mpiversion=2.2

		will build and run distgraph1 with 4 MPI processes only 
		if the MPI version is at least 2.2.

strict=bool : If bool is false, only build and run the program if 
              --enable-strictmpi was not used in configuring the test suite.
	      That is, a line such as 

              bcast 4 env=MPIR_CVAR_ENABLE_FT=1 strict=false

              Says that this test is not valid for a strict MPI implementation;
              it contains extensions to the standard, or in the case of some
	      MPICH development, MPIX routines

resultTest=proc : This is used to change the way in which the success or 
                  failure of a test is evaluated.  proc is one of several 
                  Perl subroutines defined within the runtest program.  These
                  are primarily used within the testsuite for tests programs
                  exit with expected status values or that timeouts are 
                  in fact handled.


More control over running tests
===============================

You can provide a "processes per node" argument to the run command (typically
mpiexec) with either options to "runtests" or environment variables.
The two values are

-ppnarg=string or MPITEST_PPNARG
   The string used to specify the number of processes per node.  The number
   of processes to use will be substituted for the %d in the string.  For
   example,

   export MPITEST_PPNARG="-ppn %d"

-ppn=n or MPITEST_PPNMAX
   The maximum number of processes per node.  For example

   runtests ... -ppn=2

   This allows the runtests script to ensure that the value of the
   processes per node argument does not exceed the total number of processes;
   some run commands (e.g., aprun on Cray) require that the number of
   processes per node be no greater than the total number of processes.

Note that for most systems it will be important to run the tests
multiple times, using this option to ensure that the tests that
involve more than one process are run each of the following cases:
1) Multiple MPI processes per chip (likely using shared memory to
communicate between processes)
2) MPI processes on separate chips within the same node (also likely
using shared memory between processes, but may use a different
approach to handle the NUMA nature of this case)
3) MPI processes on separate nodes (likely using the best available
interconnect).
Note, this depends on the nature of the MPI implementation; these
options make it easier to run the necessary cases.  If you run only
the first case, which is often the default case, you may not
effectively test the MPI implementation.


Adding more tests with DTPools
==============================
With the new DTPools framework integrated in the test suite, it is possible
to add more exhaustive datatype tests. To add a new test, setting the "basic
type" and "count" command line parameters is sufficient.

Starting from the basic test, many different tests for different signatures
can be generated automatically by adding a corresponding entry for the test in
the file: "basictypetest.txt," following the format defined by the file itself.
Then the tests can be generated by just running the main autogen.sh script.

In case the user wants to run a subset of all the possible tests that will be
generated, he/she can use the "DTP_RUNTIME_TYPES" environment variable to limit the
number of basic types tested by the "runtest" script. If the user further wishes
to reduce the length of the tests, he/she can also use the "DTP_NUM_OBJS" environment
variable to limit the number of datatypes tested from each pool (please refer to
the DTPools README and wiki for more information: https://wiki.mpich.org/mpich/index.php/DTPools).

DTPools also allows the user to define his/her own struct type pools. For this case
the configuration file to be used to create the executables is "structtypetest.txt".
Unlike the basic case only one test will be generated (for now). However, the user
can still define arbitrary structures in the configuration file in a flexible way.