\documentclass[dvipdfm,11pt]{article}
\usepackage[dvipdfm]{hyperref} % Upgraded url package
\parskip=.1in

\begin{document}
\markright{MPICH Logging}
\title{MPICH Logging\\
Version 0.1\\
DRAFT of \today\\
Mathematics and Computer Science Division\\
Argonne National Laboratory}

\author{David Ashton}


\maketitle

\cleardoublepage

\pagenumbering{arabic}
\pagestyle{headings}


\section{Introduction}
\label{sec:introduction}

This manual assumes that MPICH has already been installed.  For
instructions on how to install MPICH, see the MPICH Installer's Guide,
or the README in the top-level MPICH directory.  This manual will
explain how the internal logging macros are generated and how the user
can generate log files viewable in Jumpshot.


\section{Configuring mpich to create log files}
\label{sec:configuring}

When users run configure they can specify logging options.  There are three 
configure options to control logging.

\begin{description}
\item[\texttt{--enable-timing=<timing\_type>}]\mbox{}\\
Add this option to enable timing.  The two options for timing\_type are 
\texttt{log} and \texttt{log\_detailed}.  The \texttt{log} option will log 
only the MPI functions.  The 
\texttt{log\_detailed} will log every function in mpich.  This option gives 
fine grained logging information and also creates large log files.  It must 
be used in conjunction with a timer-type that can log very short intervals 
on the order of 100's of nanoseconds.

\item[\texttt{--with-logging=<logger>}]\mbox{}\\
Specify the logging library to use.  Currently the only logger option is \texttt{rlog}.

\item[\texttt{--enable-timer-type=<timer\_type>}]\mbox{}\\
Specify the timer type.  The options are
\begin{itemize}
\item \texttt{gethrtime} -
Solaris timer (Solaris systems only)
\item \texttt{clock\_gettime} -
Posix timer (where available)
\item \texttt{gettimeofday} -
Most Unix systems
\item \texttt{linux86\_cycle} -
Linux x86 cycle counter*
\item \texttt{linuxalpha\_cycle} -
Like linux86\_cycle, but for Linux Alpha*
\item \texttt{gcc\_ia64\_cycle} -
IA64 cycle counter*
\end{itemize}
* Note that CPU cycle counters count cycles, not elapsed time.
Because processor frequencies are variable, especially with modern
power-aware hardware, these are not always reliable for timing and so
should only be used if you're sure you know what you're doing.

\end{description}

Here is an example:
\begin{verbatim}
mpich/configure
    --enable-timing=log
    --with-logging=rlog
    --enable-timer-type=gettimeofday
    ...
\end{verbatim}

\section{Generating log files}
\label{sec:genlogs}
Run your mpi application to create intermediate \texttt{.irlog} files.

\begin{verbatim}
mpicc myapp.c -o myapp
mpiexec -n 3 myapp
\end{verbatim}
There will be .irlog files created for each process:
\begin{verbatim}
log0.irlog
log1.irlog
log2.irlog
\end{verbatim}

\section{RLOG tools}
\label{sec:tools}
For performance reasons each process produces a local intermediate log file 
that needs to be merged into a single rlog file.  Use the rlog tools to merge 
the \texttt{.irlog} files into an \texttt{.rlog} file.  The rlog tools are 
found in \texttt{mpich\_build/src/util/logging/rlog}. Currently they are not 
copied to the install directory.

\begin{description}
\item[\texttt{irlog2rlog}]\mbox{}\\
This tool combines the intermediate \texttt{.irlog} files into a single 
\texttt{.rlog} file. The usage is: ``\texttt{irlog2rlog outname.rlog 
input0.irlog input1.irlog ...}'' A shortcut is provided: ``\texttt{irlog2rlog 
outname.rlog <num\_files>}''.  Execute \texttt{irlog2rlog} without any 
parameters to see the usage options.

\item[\texttt{printrlog}]\mbox{}\\
This tool prints the contents of an \texttt{.rlog} file.

\item[\texttt{printirlog}]\mbox{}\\
This tool prints the contents of an \texttt{.irlog} file.
\end{description}

Continuing the example from the previous section:
\begin{verbatim}
irlog2rlog myapp.rlog 3
\end{verbatim}
will convert \texttt{log0.irlog}, \texttt{log1.irlog} and \texttt{log2.irlog} 
to \texttt{myapp.rlog}.

\section{Viewing log files}
This section describes how to view a log file

\texttt{.rlog} files can be printed from a command shell using the 
\texttt{printrlog} tool but the more interesting way to view the log files 
is from Jumpshot.  Jumpshot displays slog2 files and has a built in converter 
to convert \texttt{.rlog} files to \texttt{.slog2} files.  Start Jumpshot and 
open your \texttt{.rlog} file.  Jumpshot will ask you if you want to convert
the file and you say yes.

\section{Logging state code generation}
\label{sec:genstates}

This section can be skipped by users.  It describes the internal scripts used
to develop the logging macros.

This is how the \texttt{maint/genstates} script works:

\begin{enumerate}
\item \texttt{autogen.sh} creates \texttt{genstates} from 
\texttt{genstates.in} replacing \texttt{@PERL@} with the appropriate path to
perl and then runs \texttt{genstates}.

\item \texttt{genstates} finds all \texttt{.i}, \texttt{.h} and \texttt{.c} files 
in the mpich directory tree, searches for \texttt{\_STATE\_DECL} in each 
file and builds a list of all the MPID\_STATEs.  It validates that the states
start in a \texttt{\_STATE\_DECL} statement, followed by a \texttt{FUNC\_ENTER}
statement, and then at least one \texttt{FUNC\_EXIT} statement.  Errors are printed
out if the code does not follow this format except for macros.  State declarations
in macros are assumed to be correct.

\item \texttt{genstates} finds all the \texttt{describe\_states.txt} 
files anywhere in the mpich tree.  \texttt{describe\_states.txt} files are 
optional and are used to set the output name of the state and its associated 
color.

\item The \texttt{describe\_states.txt} file format is this:
\begin{verbatim}
MPID_STATE_XXX <user string for the state> <optional rgb color>
\end{verbatim}
 Here is an example line:
\begin{verbatim}
 MPID_STATE_MPI_SEND MPI_Send 0 0 255
\end{verbatim}
If you don't specify a state in a \texttt{describe\_states.txt} file then
the state user name will be automatically created by stripping off the 
\texttt{MPID\_STATE\_} prefix and the color will be assigned a random value.

\item \texttt{genstates} ouputs \texttt{mpich/src/include/mpiallstates.h} 
with this \texttt{enum} in it:
\begin{verbatim}
enum MPID_TIMER_STATE
{
    MPID_STATE_XXX,
    ...
};
\end{verbatim}

\item \texttt{genstates} outputs 
\texttt{mpich/src/util/logging/describe\_states.c} with the
\texttt{MPIR\_Describe\_timer\_states()} function in it.  Currently, only 
the rlog version of \texttt{MPIR\_Describe\_timer\_states()} is generated.

\end{enumerate}

\end{document}