\documentclass[11pt]{article}
\usepackage{times} % Necessary because Acrobat can't handle fonts properly
\let\file=\texttt
\let\code=\texttt
\let\program=\code
\usepackage{epsf}
\setlength\textwidth{6.5in}
\setlength\oddsidemargin{0in}
\setlength\evensidemargin{0in}
\setlength\marginparwidth{0.7in}
\def\bw{\texttt{\char`\\}}
% Add a discussion macro for parts of the paper that need discussion
\newenvironment{discussion}{\begin{quotation}\centerline{\textbf{Discussion}}}{\ifvmode\else\par\fi\centerline{\textbf{End
of Discussion}}\end{quotation}}
\newcommand\onehalf{\ifmmode {\scriptstyle {\scriptstyle 1\over
\scriptstyle 2}} \else $\onehalf$ \fi}
\renewcommand{\floatpagefraction}{0.9}
\begin{document}
\title{{\bf Process Management in MPICH}\\[.2in] DRAFT 2.1}
\author{The MPICH Team\\Argonne National Laboratory}
\maketitle
\begin{abstract}
In this note we describe a process management interface that can be
used by MPI implementations and other parallel processing libraries
yet be independent of both. We define the specific interface we are
developing, called PMI (Process Manager Interface) in the context of
MPICH. We describe the interface itself and a number of
implementations. We show how an MPI implementation built on MPICH
can use PMI to make itself independent of the environment in which it
executes, and how a process management environment can support MPI
implementations based on MPICH.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
This informal paper is intended to be useful for those using MPICH as the
basis of their own MPI implementations, or providing a process
management environment which will run MPI jobs that are linked against
MPICH itself or an MPICH-derived MPI implementation. At the time of
writing, this audience potentially includes groups at IBM (for BG/L), Livermore
(SLURM process management), Cray (MPI implementation for Red Storm, with
YOD), Myricom (MPICH-GM), Viridian (PBSPro), and Globus/Teragrid
implementors (MPICH-G2). Others are welcome to contribute suggestions.
This paper is incomplete in the sense that it describes an interface
still being defined. The part that is currently in use in the MPI-1
part of MPICH has proved adequate for our needs and has several
implementations. We will refer to it as ``Part 1.'' That part of the
interface that is required for the dynamic process management part of
MPI-2 is still under development, and we will refer to it here as ``Part
2.'' Most of this paper is about Part 1, which is all that is necessary
to support an MPI implementation that does not include dynamic process
management.
We describe the problem we are trying to solve in
Section~\ref{sec:problem}, the approach we have taken so far in MPICH
in Section~\ref{sec:approach}, the PMI interface itself in
Section~\ref{sec:interface} as it has been defined so far, and
implementations in Section~\ref{sec:implementing}. In
Section~\ref{sec:implications} we outline the implications for those who
are collaborating with us in various ways related to the MPICH project.
\section{The Problem}
\label{sec:problem}
The problem this paper addresses is how to provide the processes of a
parallel job with the information they need in order to interact with
the process management environment where necessary, in particular to set
up communication with one another. In many cases such information is
partly provided by the systems software that actually starts processes,
and also partly provided by the processes themselves, in which case th
process management system must aid in the dissemination of this
information to other processes. A classic example occurs when a process
in a TCP implementation of MPI acquires a port on which it can be
contacted, and then must notify other processes of this port so that
they can establish MPI communication with this process by connecting to
the port.
Traditionally, parallel programming libraries, such as MPI
implementations, have been integrated with process management mechanisms
(LAM, MPICH1 with ch\_p4 device, POE) in order to solve this problem.
After a preliminary exploration of separating the library from the
process manager in MPICH-1 with the {\tt ch\_p4mpd} device we have decided to
update the interface and then commit to this approach in MPICH. We are
motivated by the challenges of implementing MPI-2 in a
system-independent way, but many of the ideas here might prove useful in
a non-MPI environment as well, either for other parallel libraries (such
as Global Arrays, GP-SHMEM, or GASNet) or language-based systems (such
as UPC, Co-Array Fortran, or Titanium).
The problem to be addressed has several components:
\begin{itemize}
\item Conveying to processes in a parallel job the information they need
to establish communication with one another. To focus the discussion,
we assume that such communication is needed for implementing MPI.
Thus this information could include hosts, ports, interfaces,
shared-memory keys, and other information.
\item MPI-2 dynamic process management functions require extra support
for implementing {\tt MPI\_Comm\_Spawn}, {\tt MPI\_Comm\_\{Connect/Accept\}},
{\tt MPI\_\{Publish/Lookup\}\_name}, etc.
\item The interface should be simple and straightforward, particularly
in the absence of dynamic process management. MPICH will implement
dynamic process management, but some other MPI implementations may
not.
\item The interface must allow a scalable implementation for
performance-critical operations. In environments with even only hundreds
of processes, serial algorithms will be inappropriate.
\end{itemize}
An earlier, similar version of this approach was described
in~\cite{butler-lusk-gropp:mpd-parcomp}, where the interface is called
BNR, incorporated in the {\tt ch\_p4mpd} device in the original MPICH. PMI
represents an evolution of that interface and its implementation in MPICH.
Note that existing process managers often do a scalable job of starting
processes, and this part of existing systems can be kept. What is
sometimes lacking is a way of conveying the communication-establishment
information. Although the approach we take in the implementations below
is to combine the process startup and information exchange functionality
in a single interface, a different implementation could separate these,
using an existing process-startup mechanism and adding a new component
to implement the other parts of the interface. One approach along these
lines is outlined in Section~\ref{sec:adding}.
\section{Our Approach}
\label{sec:approach}
The approach we have taken to the problem is to define a Process
Management Interface (PMI). The MPICH implementation of MPI will be
implemented in terms of PMI rather than in terms of any particular
process management environment. Multiple implementations of PMI will
then be possible, independently of the MPICH implementation of MPI.
The key to a good design for PMI is to specify it in a way that allows for
scalable implementation without dictating any details of the
implementations. This has worked out well so far, and in
Section~\ref{sec:implementing} we describe a number of quite different
implementations of the interfaces described in Section~\ref{sec:interface}.
\section{The PMI Interface}
\label{sec:interface}
We present the interface in two parts. The first part is sufficient for
the implementation of MPI-1 and many parts of MPI-2. The second part is
required for implementing the dynamic process management part of MPI-2.
MPICH is now using the first part, with multiple PMI implementations,
so we consider it relatively final at this point. Since we have not yet
implemented the dynamic process management functions in MPICH, some
evolution of Part 2 of PMI may take place as we do so.
The fundamental idea of Part 1 is the {\em key-value space}, or KVS,
containing a set of (key, value) pairs of strings. Processes acquire
access to one or more KVS's through PMI and can perform {\tt put/get}
operations on them. Synchronization is defined in a scalable way via
the barrier operation, so that processes can be assure that the necessary
puts have been done before attempting the corresponding gets.
Thus the PMI interface (Part 1) consists of {\tt put/get/barrier} operations
together with housekeeping operations for managing the KVS's. For
implementation of MPI-1, a single KVS, the default KVS for processes
started at the same time, is sufficient, but multiple KVS's will be
useful when we consider Part 2 and dynamic process management.
\subsection{Part 1: Basic PMI Routines}
\label{sec:part1}
Part 1 of the interface is invoked in performance-critical parts of the
MPI implementation, both during initialization and connection setup.
Thus it is critical that this part of the interface allow scalable
implementation. We accomplish this through the semantics of the {\tt
put/get/barrier}, since the only synchronizing operation is the
collective {\tt barrier}, which is can have a scalable implementation.
The {\tt commit} operation allowsl batching of {\tt put} operations for
improved performance.
Part 1 has two subparts: firstly, the functions associated with the process group
being started, and thus already implemented in some way in any MPI
implementation, and secondly, the functions associated with managing the
keyval spaces, used for communicating setup information.
\begin{small}
\begin{verbatim}
/* PMI Group functions */
int PMI_Init( int *spawned ); /* initialize PMI for this process group
The value of spawned indicates whether
this process was created by
PMI_Spawn_multiple. */
int PMI_Initialized( void ); /* Return true if PMI has been initialized */
int PMI_Get_size( int *size ); /* get size of process group */
int PMI_Get_rank( int *rank ); /* get rank in process group */
int PMI_Barrier( void ); /* barrier across processes in process group */
int PMI_Finalize( void ); /* finalize PMI for this process group */
/* PMI Keyval Space functions */
int PMI_KVS_Get_my_name( char *kvsname ); /* get name of keyval space */
int PMI_KVS_Get_name_length_max( void ); /* maximum name size */
int PMI_KVS_Get_key_length_max( void ); /* maximum key size */
int PMI_KVS_Get_value_length_max( void ); /* maximum value size */
int PMI_KVS_Create( char *kvsname ); /* make a new one, get name */
int PMI_KVS_Destroy( const char *kvsname ); /* finish with one */
int PMI_KVS_Put( const char *kvsname, const char *key,
const char *value); /* put key and data */
int PMI_KVS_Commit( const char *kvsname ); /* block until all pending put
operations from this process
are complete. This is a
process local operation. */
int PMI_KVS_Get( const char *kvsname, const char *key, char *value);
/* get value associated with key */
int PMI_KVS_iter_first(const char *kvsname, char *key, char *val);
int PMI_KVS_iter_next(const char *kvsname, char *key, char *val);
/* loop through the pairs in the kvs */
\end{verbatim}
\end{small}
A scalable implementation of Part 1 of PMI could probably use existing software
for the group functions, and add some new functionality to support the
KVS-related functions. One possible implementation is suggested below
in Section~\ref{sec:adding}.
\paragraph{Notes}
\begin{itemize}
\item The above routines (Part 1) are all that is needed for an
implementation of MPI-1 and most of MPI-2. Part 2 is only needed to
support the MPI functions defined in the Dynamic Process Management
section of the MPI Standard.
\item Similarly, multiple KVS's are only really needed in the dynamic
process management case. An initial implementation could omit {\tt
PMI\_KVS\_Create} and {\tt PMI\_KVS\_Destroy}. The iterators {\tt
int PMI\_KVS\_iter\_first} and {\tt int PMI\_KVS\_iter\_next} are used to
transfer KVS's in grid environments, and could also be omitted from
some implementations.
\item The {\tt spawned} argument to {\tt PMI\_Init} is necessary to
implement MPI-2 functionality, in particular {\tt MPI\_Get\_parent}.
In a PMI implementation that does not support dynamic process
management, it can always just return a pointer to 0.
\item The {\tt PMI\_Commit} exists so that in case {\tt PMI\_Put} is an
expensive operation, involving communication with an external process,
several {\tt PMI\_Put}s can be batched locally and only sent off when
the {\tt PMI\_Commit} is done.
\item The notion of KVS is reminiscent of Linda, in which processes
execute {\tt read} and {\tt write} operations on a shared ``tuple
space''. Why not use the Linda interface? The reason is scalability.
Linda implements a blocking {\tt read}, in which the calling process blocks
until data with the requested key is put into the tuple space by
another process. While this is a convenient synchronizing operation,
and could in theory be used here, it would not be scalable. Note that
in PMI, there is no point-to-point communication. The only
synchronization operation is the {\tt PMI\_Barrier}, which can have a
variety of scalable implementations, depending on the environment.
\end{itemize}
\textbf{To Do}: Provide minimum sizes for the various strings. Here
is a proposal.
The minimum sizes of the names and values stored will depend on the
MPI implementation. The following limits will work with most
implementations:
\begin{description}
\item[kvsname]16
\item[key]32
\item[value]64
\item[Number of keys]Number of processes in an MPI program (size of
\texttt{MPI\_COMM\_WORLD} in an MPI-1 program)
\item[Number of groups]Number of separate \texttt{MPI\_COMM\_WORLD}s
managed by the process manager. For an single MPI-1 code, this is one.
\end{description}
\subsection{Part 2: Advanced PMI Routines}
\label{sec:part2}
This part of PMI is still under development. If one assumes that the
dynamic process management functions in MPI-2 are not performance
critical, then the requirements for efficiency and scalability of these
operations are less crucial, although we expect MPI\_Comm\_Spawn to be
scalably implemented, at least to compete with an original {\tt mpiexec}.
\begin{small}
\begin{verbatim}
/* PMI Process Creation functions */
int PMI_Spawn_multiple(int count, const char *cmds[], const char **argvs[],
const int *maxprocs, const void *info, int *errors,
int *same_domain, const void *preput_info);
int PMI_Spawn(const char *cmd, const char *argv[], const int maxprocs,
char *spawned_kvsname, const int kvsnamelen );
/* parse PMI implementation specific values into an info object that can
then be passed to PMI_Spawn_multiple. Remove PMI implementation
specific arguments from argc and argv
*/
int PMI_Args_to_info(int *argcp, char ***argvp, void *infop);
/* Other PMI functions to be defined as necessary for other parts of
dynamic process management */
\end{verbatim}
\end{small}
\section{Typical Usage}
\label{sec:usage}
In this section we give an example of typical usage of the PMI interface
in MPICH. In the {\tt CH3\_TCP} device used on Linux clusters, TCP is
used to support MPI communication. Connections between processes are
established by the normal socket {\tt connect}/{\tt accept} mechanism.
For this to work, before the first {\tt MPI\_SEND} from one process to
another, one process must have acquired a port from the operating system
and be listening on it with the normal {\tt socket}/{\tt bind}/{\tt listen}
sequence. The other process, typically on a separate host, will execute
the corresponding {\tt socket}/{\tt connect} sequence, at which time the
first process will issue an {\tt accept}, establishing the TCP
connection. Since we don't use reserved ports, the first process must
advertise in some way the port it is listening on. Since for the sake
of scalability and rapid startup we don't establish these connections
until they are needed, the {\tt connect} operation is not executed until
the socket is needed, typically the first time a process issues an {\tt
MPI\_SEND}. At this point the MPICH implementation must determine from the
MPI rank of the destination process which host, and which port on that
host, to connect to in order to establish the connection.
PMI is the mechanism by which the first process advertises its host and
listening port, keyed by rank, and the the second process finds out this
information. Thus the sequence of events during {\tt MPI\_Init} goes
like this:
\begin{enumerate}
\item During {\tt MPI\_Init}, each process calls {\tt PMI\_Init}, in
order to perform whatever initialization is needed by the PMI
implementation.
\item Still in {\tt MPI\_Init}, each process calls {\tt PMI\_Get\_rank}
to find out its rank in the MPI job.
\item Each process executes {\tt gethostname} to find out its host and
{\tt socket}/{\tt bind} to obtain a port.
\item Each process creates a key from its rank and a value for that key
from its host and port. We actually use two pairs, using keys {\tt
P<rank>-hostname} and {\tt P<rank>-port}.
\item Each process does a {\tt PMI\_KVS\_Put} to put its (key, value)
pairs into the default KVS. It may deposit other information with
other calls to {\tt PMI\_KVS\_Put}. It does a {\tt PMI\_Commit} to
flush all of the (key, value) pairs to the KVS.
\item All processes execute {\tt PMI\_Barrier} to synchronize. It is
assumed that this operation is implemented in a scalable way. Note
that MPI communication is not available yet, so this is not an {\tt
MPI\_Barrier}. We are still inside {\tt MPI\_Init}.
\end{enumerate}
At this point the only non-local communication that has taken place is
the barrier. Now, each process can exit from {\tt MPI\_INIT}, having
made available the information that only it knows (the listening port).
When a process executes any form of {\tt MPI\_SEND}, the
implementation can check to see if a connection to the destination
process already exists, and if not, use the rank to create the
appropriate key and do {\tt PMI\_KVS\_Get} to find the host and port of the
destination process and do the {\tt connect}. We currently keep these
sockets open for the rest of the job, but there is nothing to preclude
closing and reopening them as needed.
The above sequence is of course not the only way to use the PMI
interface, but it constitutes a typical example of its use. Note that
even if more information is conveyed in this way, the actual size of the
KVS is not anticipated to be large. Room for a few (key, value) pairs
for each process is all that is necessary.
\section{Implementing PMI}
\label{sec:implementing}
In this section we describe some implementations of PMI that we are
using, together with a design for one we are not. Although this note is
about the interface itself and not any specific implementation, it might
be useful to understand the alternatives that we have explored and are
in current use. Also, the very existence of multiple implementations
demonstrates that PMI is a real interface with a purpose, not just a
design for part of MPICH. All implementations are distributed with
MPICH, as described at the end of Section~\ref{sec:implications}.
It is useful to think of a PMI implementation as having two parts: a
{\em client\/} side and a {\em server\/} side. The client side is the
direct implementation of the PMI functions defined above, linked together
with the MPI library in the application's executable. In some cases this
part of the implementation communicates with other processes not part of
the application; we call these processes the server side of the PMI
implementation. As we shall see, the server side may not exist at all,
or be part of the client side; multiple architectures for PMI
implementations already exist. In the following subsection we describe some
existing client-side PMI implementations. In the next section we
describe some server-side implementations. Currently most of these are
part of the MPICH distribution~\cite{mpich-web-page}.
\subsection{Client Side}
\label{sec:client-side}
We currently are using three separate implementations of the client side of PMI.
\begin{description}
%% \item[uni] is a stub used for debugging. It assumes that there is only one
%% process and so needs to provide no services.
\item[simple] is our primary implementation for Unix systems. It
assumes that a socket (the PMI socket) has been created that can be
used to exchange commands with the server side of the implementation.
It is used by multiple server implementations, as described below.
\end{description}
\subsection{Server Side}
\label{sec:server-side}
One can think of the server side of a PMI implementation as that part of
a process management system that supports the client. Currently several
are in use or under development.
\begin{description}
\item[forker] implements the server side of the ``simple'' client side
described above. It is used primarily for debugging, although it can
also be used in production on SMP's. It consists of an {\tt mpiexec}
script that simply forks the parallel processes after setting up the
PMI socket. Thus all processes must be on the same machine.
\item[Remshell] uses {\tt rsh} or {\tt ssh} to start the processes from
the {\tt mpiexec} process, then they connect back to exchange the
keyval information. This illustrates the combination of an old ({\tt
rsh}) process startup mechanism with a new data-exchange mechanism.
\end{description}
\subsection{Combined Client and Server}
\label{sec:combined}
The MPICH-G2~\cite{karonis02:mpich-g2} implementation of MPI illustrates
yet another approach. MPICH-G2 is built on MPICH1 and thus uses the BNR
interface, but the underlying principles are the same. In MPICH-G2, the
{\tt put} operations are local, and the {\tt barrier} operation is a
global all-to-all-exchange, implemented in a scalable way. Then the
{\tt get}s can be done without further communication.
\subsection{Adding a PMI Module to an Existing Process Starter}
\label{sec:adding}
In the implementations listed above, we have combined the PMI
implementation, particularly the server side, with a process startup
mechanism being implemented at the same time. Some systems, such as
SLURM, may already have scalable methods in place for starting processes
and might be looking for the simplest way to add PMI capabilities.
Although the best approach is likely to be to incorporate PMI
server-side capabilities into the process starter, it may be that the
following approach, though less scalable, might be serviceable:
\begin{enumerate}
\item At the time each process of the MPI job is started, it is passed
its rank and the size of the job in an environment variable, since
these are things the process manager knows. This could be used to
implement {\tt PMI\_Get\_rank} and {\tt PMI\_Get\_size}. (Actually
these values would probably be read from the environment during {\tt
PMI\_Init} and cached.)
\item At the time the job is started, a separate ``KVS server'' process
would be forked to hold all KVS data.
\item All processes would send their {\tt PMI\_KVS\_Put} data to this
server. Use of UDP rather than TCP would probably help with the
obvious scalability problem that this server would receive data from
each process in the job at approximately the same time.
\item The {\tt PMI\_Barrier} would be implemented in the server with a
simple counter.
\item Data requested by {\tt PMI\_KVS\_Get} would come from the server.
\item A variation would be to have all the data broadcast at the time of
the barrier, so that subsequent gets would be local.
\end{enumerate}
This mechanism is not intrinsically scalable to thousands of nodes,
which is why we are not using it. However, it might scale farther than
a few hundred nodes, and be a rather straightforward addition to an
existing process startup mechanism.
\section{Resource Registration}
\label{sec:register}
There are some resources that a program may need to allocate that the
program cannot guarantee will be released when the program exits,
particularly if the program exits as the result of an error or an
uncatchable signal. These resources include other processes, SYSV
shared memory segments and semaphores, and temporary files. The
routines in this section allow the program to notify the process
manager of these resources and provide a general way for the process
manager to free them when the program exits.
If the process manager does not provide these functions, then there
are several options:
\begin{enumerate}
\item The calls can be ignored. The program will do its best to free
these resources when it exits. This may include setting a cleanup
handler on the catchable signals that normally cause an abort.
Note that in this case the registration routine must retrun an error
so that the application knows that it must handle this itself.
\item The calls can be directed to an alternate process, called a
``watchdog'', that will free the resources if the watched process
terminates abnormally.
\end{enumerate}
Note that this interface provides a way for process managers to permit
a process to create new processes, since the processes will be
registered with the process manager.
The following is still in rough draft form
\begin{verbatim}
int PMI_Resource_register( const char *name, (void *()(void*))at_exit,
void *at_exit_extra_data,
(void *()(void *))at_abort,
void *at_abort_extra_data );
int PMI_Resource_release_begin( const char *name, int timeout );
int PMI_Resource_release_end( const char *name );
\end{verbatim}
The functions in \texttt{PMI\_Resource\_register} may need to be
command names or an enumerated list of known resources.
The release functions are split to allow the process to indicate that
it is about to release a resource and a timeout at which time the
watchdog may consider the process to have failed. For example, when
removing a SYSV shared memory segment, the following code would be used:
\begin{verbatim}
PMI_Resource_release_begin( "myipc", 10 );
shmctl( memid, IPC_RMID, NULL );
PMI_Resource_release_end( "myipc" );
\end{verbatim}
This interface still contains a small race condition: the time between
when the resource is created and when it is registered. This is a
very narrow race, so it may not be important to close it (and with
registration, much more likely resource leaks have been closed).
However, a two-phase registration process could be considered, that
would register the intent to create a resource. In the case of
failure to complete the second part of the two-phase registration, the
watchdog could try to hunt down the newly allocated resource.
\section{Topology Information}
\label{sec:topology}
The process manager often has some information about the process
topology. For example, it is likely to know about multiprocessor
nodes and may know about parallel machine layout. The routines in
this section provide a way for the process manager to communicate that
information to the program. As with the other PMI services, if the
process manager cannot provide this service, several alternatives
exist, including returning an \texttt{PMI\_ERR\_UNSUPPORTED} and using
a separate service to provide this information.
\begin{verbatim}
int PMI_Topo_type( PMI_Group group, int *kind );
int PMI_Topo_cluster_info( PMI_Group group,
int *levels, int my_cluster[],
int my_rank[] );
int PMI_Topo_mesh_info( PMI_Group group, int ndims, int dims[] );
\end{verbatim}
These routines provide information on the specified PMI group.
\texttt{PMI\_Topo\_type} gets the type of topology. The current
choices are \texttt{PMI\_TOPO\_CLUSTER}, \texttt{PMI\_TOPO\_MESH}, and
\texttt{PMI\_TOPO\_NONE}.
The other routines provide information about the cluster and mesh
topology. Other topologies can be added as necessary; these cover
most current systems.
\section{Resource Allocation on Behalf of Parallel Jobs}
\label{sec:request}
(I'm not sure that this section goes here)
In some cases, resources must be allocated before a process is
created. For example, if several processes on the same SMP node are
to share an anonymous mmap (for shared memory), this memory must be
allocated before the processes are created (strictly, before all but
the first process is created, if the first process creates the
others). The purpose of the routines in this section is to allow a
startup program, such as \texttt{mpiexec}, to describe these
requirements to the process manager before any processes are started.
Question: it may be that the only routine here is used to answer the
question ``did you give me the resource''? This leaves unanswered the
question of ``how does a device let an mpiexec know that it needs a
particular resource''?
\section{Implications for Collaborators}
\label{sec:implications}
We hope that this brief discussion has made it easier to understand what
options and opportunities exist for implementors of parallel programming
libraries or process management environments that will interact with
MPICH or MPICH-derived MPI implementations.
MPI and other library implementors are recommended to use the PMI
functions to exchange data with other processes related to the setting
up of the primary communication mechanism. MPICH does this already
for setting up TCP connections in the CH3 implementation of the
Abstract Device Interface (ADI-3). If one links with the ``simple''
implementation of the client side of the PMI implementation in MPICH,
then MPI jobs can be started by any process management environment
that implements the server side.
Process management systems, such as PBS, YOD, or SLURM, have two options
in the short run.
In the long run implementations may prefer to implement both sides
themselves, meaning that one would link one's application with a PBS- or
SLURM- or Myricom-specific object file implementing the client side.
The PMI-related code described here is available in the current MPICH
distribution~\cite{mpich-web-page}, in the {\tt
src/pmi/\{simple,uni\}} (client side) and {\tt
src/pm/forker} (server side) subdirectories.
Different process managers (the server side) and different PMI
implementations can be chosen when MPICH is configured. The default is
as if one had specified
\begin{verbatim}
configure --with-pmi=simple
\end{verbatim}
Please send questions and comments to mpich-discuss@mcs.anl.gov.
%\appendix
%\section{Wire Protocol for the Simple PMI Implementation}
%\texttt{PMI\_PORT} environment variable
%\section{Man Pages for PMI Routines}
% Use the man page generator and include the relevant files here
\bibliography{/home/MPI/allbib,paper}
\bibliographystyle{plain}
\end{document}