Copyright (c) 2017, Intel Corporation
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
 are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice,
  this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.
- Neither the name of Intel Corporation nor the names of its contributors may
  be used to endorse or promote products derived from this software without
  specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL INTEL, THE COPYRIGHT OWNER OR CONTRIBUTORS
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

EXPORT LAWS: THIS LICENSE ADDS NO RESTRICTIONS TO THE EXPORT LAWS OF YOUR
JURISDICTION. It is licensee's responsibility to comply with any export
regulations applicable in licensee's jurisdiction. Under CURRENT (May 2000)
U.S. export regulations this software is eligible for export from the U.S.
and can be downloaded by or otherwise exported or reexported worldwide EXCEPT
to U.S. embargoed destinations which include Cuba, Iraq, Libya, North Korea,
Iran, Syria, Sudan, Afghanistan and any other country to which the U.S. has
embargoed goods and services.

This package contains SHMEM applications and packages for testing and
evaluation.


MPICH_PREFIX ENVIRONMENT VARIABLE:
----------------------------------

The SHMEM samples require an MPI implementation to run. Sandia
OpenSHMEM requires a Hydra* based MPI to run. The select_mpi script
will look for MVAPICH2 directories matching either
mvapich2*hfi (first) or mvapich* (second).

If desired the MPICH_PREFIX may be exported to control the selection.
Alternatively, a .prefix file can be created in this directory with the
preferred selection.

SHMEM_PREFIX ENVIRONMENT VARIABLE:
----------------------------------

By default, Make will build these programs with Sandia OpenSHMEM installed
by RPM sandia-openshmem_gcc_hfi-devel. SHMEM_PREFIX is also used by the
run_* scripts to define where bin/oshrun is found.

You can override this by setting SHMEM_PREFIX in your environment. For
example:

export SHMEM_PREFIX=/my/copy_of_shmem

The run_* sample runner scripts set SHMEM_PREFIX from a file
.shmem_prefix in the current working directory if that
SHMEM_PREFIX is not already set.

USING THESE TOOLS WITH MPI-SELECTOR:
------------------------------------

SHMEM applications are started using the mpirun from an MPI
implementation such mvapich2.

In OFED (and Intel IFS), the normal method of selecting an MPI is to
use the mpi-selector commands (type "man mpi-selector-menu" and "man
mpi-selector" for details on those commands).

To use mpi-selector with these scripts, add this line to your .bashrc
file:

source /usr/src/opa/mpi_apps/get_selected_mpi.sh

COMPILING THE TEST PROGRAMS:
----------------------------

The top level Makefile will build a simple SHMEM hello_worl application,
using the default (described above) or the value of SHMEM_PREFIX (also
described above).

RUNNING THE TEST PROGRAMS:
--------------------------

A "run_*" script is provided for the applications/benchmark.  These
scripts assume the existence of a local "shmem_hosts" file [alternate name can be
exported in MPI_HOSTS if desired] (in mpirun mode)
and that mpd is already running (if you're using mvapich2 in mpd mode).
For example, to run the barrier test, simply type:

# ./run_barrier 2

The scripts log the output of the SHMEM programs to the
/usr/src/opa/shmem_apps/logs directory.