\documentclass[dvipdfm,11pt]{article}

\usepackage[dvipdfm]{hyperref} % Upgraded url package

\parskip=.1in

\begin{document}

\markright{MPICH Logging}

\title{MPICH Logging\\

Version 0.1\\

DRAFT of \today\\

Mathematics and Computer Science Division\\

Argonne National Laboratory}

\author{David Ashton}

\maketitle

\cleardoublepage

\pagenumbering{arabic}

\pagestyle{headings}

\section{Introduction}

\label{sec:introduction}

This manual assumes that MPICH has already been installed.  For

instructions on how to install MPICH, see the MPICH Installer's Guide,

or the README in the top-level MPICH directory.  This manual will

explain how the internal logging macros are generated and how the user

can generate log files viewable in Jumpshot.

\section{Configuring mpich to create log files}

\label{sec:configuring}

When users run configure they can specify logging options.  There are three 

configure options to control logging.

\begin{description}

\item[\texttt{--enable-timing=<timing\_type>}]\mbox{}\\

Add this option to enable timing.  The two options for timing\_type are 

\texttt{log} and \texttt{log\_detailed}.  The \texttt{log} option will log 

only the MPI functions.  The 

\texttt{log\_detailed} will log every function in mpich.  This option gives 

fine grained logging information and also creates large log files.  It must 

be used in conjunction with a timer-type that can log very short intervals 

on the order of 100's of nanoseconds.

\item[\texttt{--with-logging=<logger>}]\mbox{}\\

Specify the logging library to use.  Currently the only logger option is \texttt{rlog}.

\item[\texttt{--enable-timer-type=<timer\_type>}]\mbox{}\\

Specify the timer type.  The options are

\begin{itemize}

\item \texttt{gethrtime} -

Solaris timer (Solaris systems only)

\item \texttt{clock\_gettime} -

Posix timer (where available)

\item \texttt{gettimeofday} -

Most Unix systems

\item \texttt{linux86\_cycle} -

Linux x86 cycle counter*

\item \texttt{linuxalpha\_cycle} -

Like linux86\_cycle, but for Linux Alpha*

\item \texttt{gcc\_ia64\_cycle} -

IA64 cycle counter*

\end{itemize}

* Note that CPU cycle counters count cycles, not elapsed time.

Because processor frequencies are variable, especially with modern

power-aware hardware, these are not always reliable for timing and so

should only be used if you're sure you know what you're doing.

\end{description}

Here is an example:

\begin{verbatim}

mpich/configure

    --enable-timing=log

    --with-logging=rlog

    --enable-timer-type=gettimeofday

    ...

\end{verbatim}

\section{Generating log files}

\label{sec:genlogs}

Run your mpi application to create intermediate \texttt{.irlog} files.

\begin{verbatim}

mpicc myapp.c -o myapp

mpiexec -n 3 myapp

\end{verbatim}

There will be .irlog files created for each process:

\begin{verbatim}

log0.irlog

log1.irlog

log2.irlog

\end{verbatim}

\section{RLOG tools}

\label{sec:tools}

For performance reasons each process produces a local intermediate log file 

that needs to be merged into a single rlog file.  Use the rlog tools to merge 

the \texttt{.irlog} files into an \texttt{.rlog} file.  The rlog tools are 

found in \texttt{mpich\_build/src/util/logging/rlog}. Currently they are not 

copied to the install directory.

\begin{description}

\item[\texttt{irlog2rlog}]\mbox{}\\

This tool combines the intermediate \texttt{.irlog} files into a single 

\texttt{.rlog} file. The usage is: ``\texttt{irlog2rlog outname.rlog 

input0.irlog input1.irlog ...}'' A shortcut is provided: ``\texttt{irlog2rlog 

outname.rlog <num\_files>}''.  Execute \texttt{irlog2rlog} without any 

parameters to see the usage options.

\item[\texttt{printrlog}]\mbox{}\\

This tool prints the contents of an \texttt{.rlog} file.

\item[\texttt{printirlog}]\mbox{}\\

This tool prints the contents of an \texttt{.irlog} file.

\end{description}

Continuing the example from the previous section:

\begin{verbatim}

irlog2rlog myapp.rlog 3

\end{verbatim}

will convert \texttt{log0.irlog}, \texttt{log1.irlog} and \texttt{log2.irlog} 

to \texttt{myapp.rlog}.

\section{Viewing log files}

This section describes how to view a log file

\texttt{.rlog} files can be printed from a command shell using the 

\texttt{printrlog} tool but the more interesting way to view the log files 

is from Jumpshot.  Jumpshot displays slog2 files and has a built in converter 

to convert \texttt{.rlog} files to \texttt{.slog2} files.  Start Jumpshot and 

open your \texttt{.rlog} file.  Jumpshot will ask you if you want to convert

the file and you say yes.

\section{Logging state code generation}

\label{sec:genstates}

This section can be skipped by users.  It describes the internal scripts used

to develop the logging macros.

This is how the \texttt{maint/genstates} script works:

\begin{enumerate}

\item \texttt{autogen.sh} creates \texttt{genstates} from 

\texttt{genstates.in} replacing \texttt{@PERL@} with the appropriate path to

perl and then runs \texttt{genstates}.

\item \texttt{genstates} finds all \texttt{.i}, \texttt{.h} and \texttt{.c} files 

in the mpich directory tree, searches for \texttt{\_STATE\_DECL} in each 

file and builds a list of all the MPID\_STATEs.  It validates that the states

start in a \texttt{\_STATE\_DECL} statement, followed by a \texttt{FUNC\_ENTER}

statement, and then at least one \texttt{FUNC\_EXIT} statement.  Errors are printed

out if the code does not follow this format except for macros.  State declarations

in macros are assumed to be correct.

\item \texttt{genstates} finds all the \texttt{describe\_states.txt} 

files anywhere in the mpich tree.  \texttt{describe\_states.txt} files are 

optional and are used to set the output name of the state and its associated 

color.

\item The \texttt{describe\_states.txt} file format is this:

\begin{verbatim}

MPID_STATE_XXX <user string for the state> <optional rgb color>

\end{verbatim}

 Here is an example line:

\begin{verbatim}

 MPID_STATE_MPI_SEND MPI_Send 0 0 255

\end{verbatim}

If you don't specify a state in a \texttt{describe\_states.txt} file then

the state user name will be automatically created by stripping off the 

\texttt{MPID\_STATE\_} prefix and the color will be assigned a random value.

\item \texttt{genstates} ouputs \texttt{mpich/src/include/mpiallstates.h} 

with this \texttt{enum} in it:

\begin{verbatim}

enum MPID_TIMER_STATE

{

    MPID_STATE_XXX,

    ...

};

\end{verbatim}

\item \texttt{genstates} outputs 

\texttt{mpich/src/util/logging/describe\_states.c} with the

\texttt{MPIR\_Describe\_timer\_states()} function in it.  Currently, only 

the rlog version of \texttt{MPIR\_Describe\_timer\_states()} is generated.

\end{enumerate}

\end{document}