% -*- Mode: latex; -*-

\documentclass[dvipdfm,11pt]{article}

\usepackage[dvipdfm]{hyperref} % Upgraded url package

\parskip=.1in

% Formatting conventions for contributors

% 

% A quoting mechanism is needed to set off things like file names, command

% names, code fragments, and other strings that would confuse the flow of

% text if left undistinguished from preceding and following text.  In this

% document we use the LaTeX macro '\texttt' to indicate such text in the

% source, which normally produces, when used as in '\texttt{special text}',

% the typewriter font.

% It is particularly easy to use this convention if one is using emacs as

% the editor and LaTeX mode within emacs for editing LaTeX documents.  In

% such a case the key sequence ^C^F^T (hold down the control key and type

% 'cft') produces '\texttt{}' with the cursor positioned between the

% braces, ready for the special text to be typed.  The closing brace can

% be skipped over by typing ^e (go to the end of the line) if entering

% text or ^C-} to just move the cursor past the brace.

% LaTeX mode is usually loaded automatically.  At Argonne, one way to 

% get several useful emacs tools working for you automatically is to put

% the following in your .emacs file.

% (require 'tex-site)

% (setq LaTeX-mode-hook '(lambda ()

%                        (auto-fill-mode 1)

%                        (flyspell-mode 1)

%                        (reftex-mode 1)

%                        (setq TeX-command "latex")))

%

% When updating the version number of MPICH, make sure that you change *all*

% instances of the version.  Search for mpich- and check each match.  Some

% are within a verbatim statement and will need to be changed for each update.

%

\begin{document}

\markright{MPICH Installer's Guide}

\title{{\bf MPICH Installer's Guide}\thanks{This work was supported by the

    Mathematical, Information, and Computational Sciences Division

    subprogram of the Office of Advanced Scientific Computing Research,

    SciDAC Program, Office of Science, U.S. Department of Energy, under

    Contract DE-AC02-06CH11357.}\\

  Version %MPICH_VERSION%\\

  Mathematics and Computer Science Division\\

  Argonne National Laboratory}

\author{

Abdelhalim Amer \and Pavan Balaji \and Wesley Bland \and William Gropp \and

Yanfei Guo \and Rob Latham \and Huiwei Lu \and Lena Oden \and Antonio J. Pe\~na

\and Ken Raffenetti \and Sangmin Seo \and Min Si \and Rajeev Thakur \and

Junchao Zhang \and Xin Zhao

}

\maketitle

\cleardoublepage

\pagenumbering{roman}

\tableofcontents

\clearpage

\pagenumbering{arabic}

\pagestyle{headings}

%% Here is a basic outline for the document.   (Bill's original outline)

%% 0. Quick start with ``best practices''.  Each step has a reference to

%% more detailed information later in the document.

%% 1. Acquiring and unpacking.  Using a ``fast'' directory location and

%%    VPATH

%% 1a. Reporting problems

%% 2. Choosing a device (defer a detailed discussion of each until later)

%% 3. configure, make, and install.  Always use --prefix 

%% show only basic options for configure

%%    3a. Optional include of device-specific information

%%    3b. Optional include of pm-specific information

%%    3c. Optional ``fast'' version

%%    3d. Shared libraries

%% 4. Testing and benchmarking

%% 4a. make testing

%% 4b. Getting, building, and using mpptest and netpipe

%% 5. Special options

%% 6. Troubleshooting

%% Appendix:

%% A. Summary of configure options (particularly the enable and with options)

\section{Introduction}

\label{sec:intro}

This manual describes how to obtain and install MPICH, the MPI

implementation from Argonne National Laboratory.  (Of course, if you are

reading this, chances are good that you have already obtained it and

found this document, among others, in its \texttt{doc} subdirectory.)

This \emph{Guide} will explain how to install MPICH so that you and others can use it to

run MPI applications.  Some particular features are different

if you have system administration privileges (can become ``root'' on a

Unix system), and these are explained here.  It is not necessary to have

such privileges to build and install MPICH.  In the event of problems,

send mail to \texttt{discuss@mpich.org}.  Once MPICH is

installed, details on how to run MPI jobs are covered in the \emph{MPICH

User's Guide}, found in this same \texttt{doc} subdirectory.

MPICH has many options.  We will first go through a recommended,

``standard'' installation in a step-by-step fashion, and later describe

alternative possibilities. 

\section{Quick Start}

\label{sec:quick}

In this section we describe a ``default'' set of installation steps.

It uses the default set of configuration options, which builds the

\texttt{nemesis} communication device and the \texttt{Hydra} process

manager, for languages C, C++, Fortran-77, and Fortran-90 (if those

compilers exist), with compilers chosen automatically from the user's

environment, without tracing and debugging options.  It uses the

\texttt{VPATH} feature of \texttt{make}, so that the build process can

take place on a local disk for speed.

\subsection{Prerequisites}

\label{sec:prerequisites}

For the default installation, you will need:

\begin{enumerate}

\item A copy of the distribution, \texttt{mpich.tar.gz}.

\item A C compiler.

\item A Fortran-77, Fortran-90, and/or C++ compiler if you wish to write

  MPI programs in any of these languages.

\item Any one of a number of Unix operating systems, such as IA32-Linux.

  MPICH is most extensively tested on Linux;  there remain some

  difficulties on systems to which we do not currently have access.  Our

  \texttt{configure} script attempts to adapt MPICH to new systems. 

\end{enumerate}

Configure will check for these prerequisites and try to work around

deficiencies if possible.  (If you don't have Fortran, you will

still be able to use MPICH, just not with Fortran applications.)

\subsection{From A Standing Start to Running an MPI Program}

\label{sec:steps}

Here are the steps from obtaining MPICH through running your own

parallel program on multiple machines.

\begin{enumerate}

\item 

Unpack the tar file.

\begin{verbatim}

    tar xfz mpich.tar.gz

\end{verbatim}

If your tar doesn't accept the z option, use

\begin{verbatim}

    gunzip -c mpich.tar.gz | tar xf -

\end{verbatim}

Let us assume that the directory where you do this is

\texttt{/home/you/libraries}.  It will now contain a subdirectory named

\texttt{mpich-%MPICH_VERSION%}.

\item

Choose an installation directory (the default is \texttt{/usr/local/bin)}:

\begin{verbatim}

    mkdir /home/you/mpich-install

\end{verbatim}

It will be most convenient if this directory is shared by all of the

machines where you intend to run processes.  If not, you will have

to duplicate it on the other machines after installation.  Actually, if

you leave out this step, the next step will create the directory for you.

\item

Choose a build directory.  Building will proceed \emph{much} faster if

your build directory is on a file system local to the machine on which

the configuration and compilation steps are executed.  It is preferable

that this also be separate from the source directory, so that the

source directories remain

clean and can be reused to build other copies on other machines.

\begin{verbatim}

    mkdir /tmp/you/mpich-%MPICH_VERSION%

\end{verbatim}

\item

Choose any configure options.  See

Section~\ref{sec:important-configure-options} for a description of the most

important options to consider.  

\item

Configure MPICH, specifying the installation directory, and running

the \texttt{configure} script in the source directory:

\begin{verbatim}

    cd  /tmp/you/mpich-%MPICH_VERSION%

    /home/you/libraries/mpich-%MPICH_VERSION%/configure \

            -prefix=/home/you/mpich-install |& tee c.txt

\end{verbatim}

where the \texttt{$\backslash$} means that this is really one line.  (On

\texttt{sh} and its derivatives, use \verb+2>&1 | tee c.txt+

instead of \verb+|& tee c.txt+).  Other configure options are

described below.  Check the \texttt{c.txt} file to make sure

everything went well.  Problems should be self-explanatory, but if not,

send \texttt{c.txt} to \texttt{discuss@mpich.org}.

The file \texttt{config.log} is created by \texttt{configure} and

contains a record of the tests that \texttt{configure} performed.  It

is normal for some tests recorded in \texttt{config.log} to fail.  

\item

Build MPICH:

\begin{verbatim}

    make |& tee m.txt       (for csh and tcsh)

      OR

    make 2>&1 | tee m.txt   (for bash and sh)

\end{verbatim}

This step should succeed if there were no problems with the preceding

step.  Check file \texttt{m.txt}. If there were problems,

do a \texttt{make clean} and then run make again with \texttt{VERBOSE=1}

\begin{verbatim}

    make VERBOSE=1 |& tee m.txt       (for csh and tcsh)

      OR

    make VERBOSE=1 2>&1 | tee m.txt   (for bash and sh)

\end{verbatim}

and then send \texttt{m.txt} and \texttt{c.txt} to 

\texttt{discuss@mpich.org}.

\item

Install the MPICH commands:

\begin{verbatim}

    make install |& tee mi.txt

\end{verbatim}

This step collects all required executables and scripts in the \texttt{bin}

subdirectory of the directory specified by the prefix argument to

configure. 

(For users who want an install directory structure compliant to

GNU coding standards (i.e., documentation files go to

\texttt{\$\{datarootdir\}/doc/\$\{PACKAGE\}}, architecture independent

read-only files go to \texttt{\$\{datadir\}/\$\{PACKAGE\}}), replace

\texttt{make install} by

\begin{verbatim}

       make install PACKAGE=mpich-<versrion>

\end{verbatim}

and corresponding installcheck step should be

\begin{verbatim}

       make installcheck PACKAGE=mpich-<version>

\end{verbatim}

Setting PACKAGE in \texttt{make install} or \texttt{make installcheck} step

is optional and unnecessary for typical MPI users.)

\item

Add the \texttt{bin} subdirectory of the installation directory to your path:

\begin{verbatim}

    setenv PATH /home/you/mpich-install/bin:$PATH

\end{verbatim}

for \texttt{csh} and \texttt{tcsh}, or 

\begin{verbatim}

    export PATH=/home/you/mpich-install/bin:$PATH

\end{verbatim}

for \texttt{bash}, and

\begin{verbatim}

    PATH=/home/you/mpich-install/bin:$PATH

    export PATH

\end{verbatim}

for \texttt{sh} (the \texttt{bash} syntax may work for more recent 

implementations of \texttt{sh}).  % export var=value is an extension and

                                  % is not in original sh

Check that everything is in order at

this point by doing

\begin{verbatim}

    which mpicc

    which mpiexec

\end{verbatim}

All should refer to the commands in the \texttt{bin} subdirectory of your

install directory.  It is at this point that you will need to

duplicate this directory on your other machines if it is not

in a shared file system. % Just having NFS doesn't mean that directories

                         % are consistencly cross-mounted.

\item Check that you can reach these machines with \texttt{ssh} or

\texttt{rsh} without entering a password.  You can test by doing

\begin{verbatim}

    ssh othermachine date

\end{verbatim}

or

\begin{verbatim}

    rsh othermachine date

\end{verbatim}

If you cannot get this to work without entering a password, you will

need to configure \texttt{ssh} or \texttt{rsh} so that this can be

done.

\item Test the setup you just created:

\begin{verbatim}

    mpiexec -f machinefile -n <number> hostname

\end{verbatim}

The machinefile contains the list of hosts you want to run the

executable on.

\begin{verbatim}

  % cat machinefile

     host1     # Run 1 process on host1

     host2:4   # Run 4 processes on host2

     host3:2   # Run 2 processes on host3

     host4:1   # Run 1 process on host4

\end{verbatim}

\item

Now we will run an MPI job, using the \texttt{mpiexec} command as specified

in the MPI standard.

As part of the build process for MPICH, a simple program to compute the value

of $\pi$ by numerical integration is created in the

\texttt{mpich-%MPICH_VERSION%/examples} directory.  If the current directory is the top

level MPICH build directory, then you can run this program with

\begin{verbatim}

    mpiexec -n 5 -f machinefile ./examples/cpi

\end{verbatim}

The \texttt{cpi} example will tell you which hosts it is running on.

There are many options for \texttt{mpiexec}, by which multiple

executables can be run, hosts can be specified, separate command-line

arguments and environment variables can be passed to different

processes, and working directories and search paths for executables

can be specified.  Do

\begin{verbatim}

    mpiexec --help

\end{verbatim}

for details. A typical example is:

\begin{verbatim}

    mpiexec -f machinefile -n 1 ./master : -n 19 ./slave

\end{verbatim}

to ensure that the process with rank 0 runs on your workstation.

The arguments between `:'s in this syntax are called ``argument sets,''

since they apply to a set of processes.  More argments are described in

the \textit{User's Guide}.

\end{enumerate}

If you have completed all of the above steps, you have successfully

installed MPICH and run an MPI example.  

%

% DO NOT REMOVE THE FOLLOWING SECTION.  IF NECESSARY, UPDATE IT.

\subsection{Selecting the Compilers}

\label{sec:compilers}

The MPICH configure step will attempt to find the C, C++, and Fortran

compilers for you, but if you either want to override the default or

need to specify a compiler that configure doesn't recognize, you can

specify them on the command line using these variables

\begin{description}

\item[CC]The C compiler.  

\item[CXX]The C++ compiler.  Use \texttt{--disable-cxx} if you do not

  want to build the MPI C++ interface

\item[F77]The Fortran 77 compiler (for the original MPI Fortran

  bindings).  Use \texttt{--disable-f77} if you do not want to build

  either the Fortran 77 or Fortran 90 MPI interfaces

\item[FC]The Fortran 90 (or later) compiler.  Use

  \texttt{--disable-fc} if you do not want to build the Fortran 90 MPI

  interfaces.  Note that in previous versions of MPICH, the variable

  name was \texttt{F90}.  As Fortran has had 3 major releases since

  Fortran 90 (95, 2003, and 2008), most tools, including those built

  with GNU autotools, have or are changing to use \texttt{FC} instead

  of \texttt{F90}.

\end{description}

For example, to select the Intel compilers instead of the GNU

compilers on a system with both, use

\begin{verbatim}

    ./configure CC=icc CXX=icpc F77=ifort FC=ifort ...

\end{verbatim}

Note the use of the same Fortran compiler, \texttt{ifort}, for both

Fortran 77 and Fortran 90; this is an increasingly common choice.

\subsection{Compiler Optimization Levels}

MPICH can be configured with two sets of compiler flags:

\texttt{CFLAGS}, \texttt{CXXFLAGS}, \texttt{FFLAGS}, \texttt{FCFLAGS}

(abbreviated as \texttt{xFLAGS}) and \texttt{MPICHLIB\_CFLAGS},

\texttt{MPICHLIB\_CXXFLAGS}, \texttt{MPICHLIB\_FFLAGS},

\texttt{MPICHLIB\_FCFLAGS} (abbreviated as \texttt{MPICHLIB\_xFLAGS})

for compilation; \texttt{LDFLAGS} and \texttt{MPICHLIB\_LDFLAGS} for linking.

All these flags can be set as part of configure command or 

through environment variables.

(\texttt{CPPFLAGS} stands for C preprocessor flags, which should NOT be set)

Both \texttt{xFLAGS} and \texttt{MPICHLIB\_xFLAGS} affect the compilation

of the MPICH libraries. However, only \texttt{xFLAGS} is appended to 

MPI wrapper scripts, \texttt{mpicc} and friends.

MPICH libraries are built with default compiler optimization, \texttt{-O2},

which can be modified by \texttt{--enable-fast} configure option.  For instance,

\texttt{--disable-fast} disables the default optimization option.

\texttt{--enable-fast=O<n>} sets default compiler optimization as

\texttt{-O<n>} (note that this assumes that the compiler accepts this

format). 

For more details of \texttt{--enable-fast}, see the output of

\texttt{configure --help}. 

Any other complicated optimization flags for MPICH libraries have

to be set throught \texttt{MPICHLIB\_xFLAGS}.  \texttt{CFLAGS} and friends

are empty by default.

For example, to build a production MPICH environment with \texttt{-O3} for all

language bindings, one can simply do

\begin{verbatim}

  ./configure --enable-fast=all,O3

\end{verbatim}

or

\begin{verbatim}

  ./configure --enable-fast=all MPICHLIB_CFLAGS=-O3 \

                                MPICHLIB_FFLAGS=-O3 \

                                MPICHLIB_CXXFLAGS=-O3 \

                                MPICHLIB_FCFLAGS=-O3

\end{verbatim}

This will cause the MPICH libraries to be built with \texttt{-O3},

and \texttt{-O3} will

not be included in the \texttt{mpicc} and other MPI wrapper script.

There are certain compiler flags that should not be used with MPICH's

configure, e.g. gcc's \texttt{-Werror} which would confuse configure and cause

certain configure tests to fail to detect the correct system features.

To use \texttt{-Werror} in building MPICH libraries, you can pass the compiler

flags during the make step through the Makefile variable,

\texttt{MPICH\_MAKE\_CFLAGS} as follows:

\begin{verbatim}

  make VERBOSE=1 MPICH_MAKE_CFLAGS="-Wall -Werror"

\end{verbatim}

(assume CC is set to gcc).  The content of \texttt{MPICH\_MAKE\_CFLAGS}

is appended to the \texttt{CFLAGS} in almost all Makefiles.

\subsection{Common Non-Default Configuration Options}

\label{sec:non-default}

A brief discussion of some of the \texttt{configure} options is found in

Section~\ref{configure-options}.  Here we comment on some of the most

commonly used options.  

% FIXME: This section needs much more work.

\subsubsection{The Most Important Configure Options}

\label{sec:important-configure-options}

\begin{description}

\item[--prefix]Set the installation directories for MPICH.  

\item[--enable-debuginfo]Provide access to the message queues for

  debuggers such as Totalview.

\item[--enable-g]Build MPICH with various debugging options.  This is

  of interest primarily to MPICH developers.  The options

\begin{verbatim}

  --enable-g=dbg,mem,log

\end{verbatim}

 are recommended in that case.

\item[--enable-fast]Configure MPICH for fastest performance at the

  expense of error reporting and other program development aids.  This

  is recommended only for getting the best performance out of proven

  production applications, and for benchmarking.

\item[--enable-shared]Build MPICH with shared libraries. 

  MPICH will try to automatically detect the type of shared library

  support required. See Section~\ref{sec:shared-libraries} for more details.

\item[--with-pm]Select the process manager.  The default is

  \texttt{hydra}; also useful are \texttt{gforker} and \texttt{remshell}.  You can build with

  all three process managers by specifying

\begin{verbatim}

  --with-pm=hydra:gforker:remshell

\end{verbatim}

\item[--with-java]Set the location of Java installation.  This option

is necessary only if the default Java installation in your PATH does

not contain a valid Java installation for Jumpshot, e.g.

\begin{verbatim}

  --with-java=/opt/jdk1.6.0

\end{verbatim}

\end{description}

\subsubsection{Using the Absoft Fortran compilers with MPICH}

\label{sec:absoft}

For best results, it is important to force the Absoft Fortran compilers to 

make all routine names monocase.  In addition, if lower case is chosen 

(this will match common use by many programs), you must also tell the the 

Absoft compiles to append an underscore to global names in order to access 

routines such as \texttt{getarg} (\texttt{getarg} is not used by MPICH but is

used in some 

of the tests and is often used in application programs).  We recommend 

configuring MPICH with the following options

\begin{verbatim}

setenv F77 f77

setenv FFLAGS "-f -N15"

setenv FCFLAGS "-YALL_NAMES=LCS -YEXT_SFX=_"

./configure ....

\end{verbatim}

\subsection{Shared Libraries}

\label{sec:shared-libraries}

To have shared libraries created when MPICH is built, specify the

following when MPICH is configured:

\begin{verbatim}

    configure --enable-shared

\end{verbatim}

\subsection{What to Tell the Users}

\label{sec:telling}

Now that MPICH has been installed, the users have to be informed of how

to use it.  Part of this is covered in the \emph{User's Guide}.  Other

things users need to know are covered here.

\section{Migrating from MPICH1}

\label{sec:migrating}

MPICH is an all-new rewrite of MPICH1.  Although the basic steps for

installation have remained the same (\texttt{configure}, \texttt{make},

\texttt{make install}), a number of things have changed.  In this

section we attempt to point out what you may be used to in MPICH1 that

are now different in MPICH.

\subsection{Configure Options}

\label{sec:configure-options}

The arguments to \texttt{configure} are different in MPICH1 and MPICH;

the \texttt{Installer's Guide} discusses \texttt{configure}.  In

particular, the newer \texttt{configure} in MPICH does not support the

\verb+-cc=<compiler-name>+ (or \texttt{-fc}, \texttt{-c++}, or

\texttt{-f90}) options.  Instead, many of the items that could be

specified in the command line to configure in MPICH1 must now be set by

defining an environment variable.  E.g., while MPICH1 allowed

\begin{verbatim}

    ./configure -cc=pgcc

\end{verbatim}

MPICH requires

\begin{verbatim}

    ./configure CC=pgcc

\end{verbatim}

Basically, every

option to the MPICH-1 configure that does not start with

\texttt{--enable} or \texttt{--with} is not available as a configure

option in MPICH.  Instead, environment variables must be used.  This is

consistent (and required) for use of version 2 GNU \texttt{autoconf}.

\subsection{Other Differences}

Other differences between MPICH1 and MPICH include the handling of

process managers and the choice of communication device.

For example, the new process managers have a new format and slightly

different semantics for the \texttt{-machinefile} option.  Assume that

you type this data into a file named \texttt{machfile}:

\begin{verbatim}

    bp400:2

    bp401:2

    bp402:2

    bp403:2

\end{verbatim}

If you then run a parallel job with this machinefile, you would expect

ranks 0 and 1 to run on bp400 because it says to run 2 processes there

before going on to bp401.  Ranks 2 and 3 would run on bp401, and rank

4 on bp402, e.g.:

\begin{verbatim}

    mpiexec -l -machinefile machfile -n 5 hostname

\end{verbatim}

produces:

\begin{verbatim}

    0: bp400

    1: bp400

    2: bp401

    3: bp401

    4: bp402

\end{verbatim}

\section{Choosing the Communication Device}

\label{sec:choose-device}

MPICH is designed to be build with many different communication devices,

allowing an implementation to be tuned for different communication fabrics.  A

simple communication device, known as ``ch3'' (for the third version of the

``channel'' interface) is provided with MPICH and is the default choice.

The ch3 device itself supports a variety of communication methods.  These are

specified by providing the name of the method after a colon in the

\texttt{--with-device} configure option.  For example,

\texttt{--with-device=ch3:sock} selects the (older) socket-base communication

method.

Methods supported by the MPICH group include:

\begin{description}

\item[ch3:nemesis]This method is our new, high performance method. It

  has been made the default communication channel starting the 1.1

  release of MPICH.  It uses shared-memory to send messages between

  processes on the same node and the network for processes between

  nodes.  Currently sockets and Myrinet-MX are supported networks.  It

  supports \texttt{MPI\_THREAD\_MULTIPLE} and other levels of thread

  safety.

\item[ch3:sock]This method uses sockets for all communications between

  processes. It supports \texttt{MPI\_THREAD\_MULTIPLE} and other

  levels of thread safety.

\end{description}

Most installations should use the default \texttt{ch3:nemesis} method

for best performance. For platforms that are not supported by nemesis,

the \texttt{ch3:sock} method is suggested.

MPICH is designed to efficiently support all types of systems.  The

\texttt{ch3:nemesis} device is the primary focus of the MPICH group,

but other research groups and computer vendors can and have developed

both their own ch3 ``channels'' as well as complete communication

``devices'' in place of ch3.

%

% Add any configure options that are specific to each method here.  Also

% any environment variables that might apply (e.g., MPICH_INTERFACE_HOSTNAME 

% or MPICH_PORT_RANGE.

%

%\subsection{Special Features of Each Communication Device}

%\subsubsection{ch3:nemesis}

\section{Installing and Managing Process Managers}

\label{sec:process-managers}

MPICH has been designed to work with multiple process managers; that

is, although you can start MPICH jobs with \texttt{mpiexec}, there are

different mechanisms by which your processes are started.  An interface

(called PMI) isolates the MPICH library code from the process manager.

Currently three process managers are distributed with MPICH

\begin{description}

\item[hydra] This is the default process manager tha natively uses the

existing daemons on the system such as ssh, slurm, pbs.

\item[gforker] This is a simple process manager that creates all

  processes on a single machine.  It is useful both for debugging and

  for running on shared memory multiprocessors.

\end{description}

\subsection{hydra} \label{sec:hydra} \texttt{hydra} is the default

process manager that launches processes using the native daemons

present on the system such as ssh, slurm, pbs, etc. To configure with

the \texttt{hydra} process manager, use

\begin{verbatim}

    configure --with-pm=hydra ...

\end{verbatim}

\subsection{gforker}

\label{sec:forker}

\texttt{gforker} is a simple process manager that runs all processes on

a single node; it's version of \texttt{mpiexec} uses the system

\texttt{fork} and \texttt{exec} calls to create the new processes.

To configure with the \texttt{gforker} process manager, use

\begin{verbatim}

    configure --with-pm=gforker ...

\end{verbatim}

\section{Testing}

\label{sec:testing}

Once MPICH has been installed, you can test it by running some of the example

programs in the \texttt{examples} directory.  A more thorough test can be run

with the command \texttt{make testing}.  This will produce a summary on

standard output, along with an XML version of the test results in 

\texttt{mpich/test/mpi}. In addition, running \texttt{make testing} from the

top-level (\texttt{mpich}) directory will run tests of the commands, such as

\texttt{mpicc} and \texttt{mpiexec}, that are included with MPICH.

Other MPI test suites are available from

\url{http://www.mcs.anl.gov/mpi/mpi-test/tsuite.html}.  As part of the MPICH

development, we run the MPICH1, MPICH, C++, and Intel test suites every night

and post the results on

\url{http://www.mpich.org/static/cron/tests/}.

Other tests are run on an occasional basis.  

% \subsection{Using the Intel Test Suite}

% \label{sec:intel}

% These instructions are local to our test environment at Argonne.

% How to run a select set of tests from the Intel test suite:

% \begin{small}

% \begin{verbatim}

% 1) checkout the Intel test suite (svn co https://svn.mcs.anl.gov/repos/mpi/test_suites/intel/trunk IntelMPITEST) (outside users

%    can access the most recent version of the test suite from the

%    test suite web page).

% 2) create a testing directory separate from the IntelMPITEST source

% directory

% 3) cd into that testing directory

% 4) run "<ITS_SRC_DIR>/configure --with-mpich=<MPICH_INSTALL_DIR>", where

% <ITS_SRC_DIR> is the path to the directory Intel test suite source (e.g.,

% /home/toonen/Projects/MPI-Tests/IntelMPITEST) and <MPICH_INSTALL_DIR> is

% the directory containing your MPICH installation

% 5) mkdir Test; cd Test

% 6) find tests in <ITS_SRC_DIR>/{c,fortran} that you are interested in

% running and place the test names in a file.  For example:

% % ( cd /home/toonen/Projects/MPI-Tests/IntelMPITEST/Test ; \

%     find {c,fortran} -name 'node.*' -print | grep 'MPI_Test' 

%     | sed -e 's-/node\..*$--' ) |& tee testlist

% Test/c/nonblocking/functional/MPI_Test

% Test/c/nonblocking/functional/MPI_Testall

% Test/c/nonblocking/functional/MPI_Testany

% Test/c/nonblocking/functional/MPI_Testsome

% Test/c/persist_request/functional/MPI_Test_p

% Test/c/persist_request/functional/MPI_Testall_p

% Test/c/persist_request/functional/MPI_Testany_p

% Test/c/persist_request/functional/MPI_Testsome_p

% Test/c/probe_cancel/functional/MPI_Test_cancelled_false

% Test/fortran/nonblocking/functional/MPI_Test

% Test/fortran/nonblocking/functional/MPI_Testall

% Test/fortran/nonblocking/functional/MPI_Testany

% Test/fortran/nonblocking/functional/MPI_Testsome

% Test/fortran/persist_request/functional/MPI_Test_p

% Test/fortran/persist_request/functional/MPI_Testall_p

% Test/fortran/persist_request/functional/MPI_Testany_p

% Test/fortran/persist_request/functional/MPI_Testsome_p

% Test/fortran/probe_cancel/functional/MPI_Test_cancelled_false

% %

% 7) run the tests using ../bin/mtest:

% % ../bin/mtest -testlist testlist -np 6 |& tee mtest.log

% %

% NOTE: some programs hang if less they are run with less than 6 processes.

% 8) examine the summary.xml file.  look for '<STATUS>fail</STATUS>' to see if

% any failures occurred.  (search for '>fail<' works as well)

% \end{verbatim}

% \end{small}

\section{Benchmarking}

\label{sec:benchmarking}

There are many benchmarking programs for MPI implementations.  Three

that we use are \texttt{mpptest}

(\url{http://www.mcs.anl.gov/mpi/mpptest}), \texttt{netpipe}

(\url{http://www.scl.ameslab.gov/netpipe}), and \texttt{SkaMPI}

(\url{http://liinwww.ira.uka.de/~skampi}).  Each of these has

different strengths and weaknesses and reveals different properties of

the MPI implementation.

In addition, the

MPICH test suite contains a few programs to test for performance artifacts in

the directory \texttt{test/mpi/perf}.  An example of a performance artifact is

markedly different performance for the same operation when performed in two

different ways.  For example, using an MPI datatype for a non-contiguous

transfer should not be much slower than packing the data into a contiguous

buffer, sending it as a contiguous buffer, and then unpacking it into the

destination buffer.  An example of this from the MPI-1 standard illustrates

the use of MPI datatypes to transpose a matrix ``on the fly,'' and one test in

\texttt{test/mpi/perf} checks that the MPI implementation performs well in

this case.

\section{All Configure Options}

\label{configure-options}

To get the latest list of all the configure options recognized by the

top-level configure, use:

\begin{verbatim}

    configure --help

\end{verbatim}

Not all of these options may be fully supported yet.  

% Fixme:  Explain some of them, at least the ones that we ask the

% users to try.

\paragraph{Notes on the configure options.}

The \texttt{--with-htmldir} and \texttt{--with-docdir} options specify the 

directories into which the documentation will be installed by \texttt{make

  install}. 

\appendix

\end{document}

%

% Comments on subclassing the document

% We can use \ifdevname ... \fi and \ifpmname ... \fi, as in

% \ifdevchiii .. \fi and \ifpmmpd ... \fi

% (these will still need to be defined)

% There should also be a way to select ``all'' in such a way that the

% document can still flow well, such as

% \ifdevall ... \else \ifdevchiii \else \ifdevmm \fi \fi \fi