|
Packit |
961e70 |
|
|
Packit |
961e70 |
This file is provided under a dual BSD/GPLv2 license. When using or
|
|
Packit |
961e70 |
redistributing this file, you may do so under either license.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
GPL LICENSE SUMMARY
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Copyright(c) 2017 Intel Corporation.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This program is free software; you can redistribute it and/or modify
|
|
Packit |
961e70 |
it under the terms of version 2 of the GNU General Public License as
|
|
Packit |
961e70 |
published by the Free Software Foundation.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This program is distributed in the hope that it will be useful, but
|
|
Packit |
961e70 |
WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
Packit |
961e70 |
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
Packit |
961e70 |
General Public License for more details.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Contact Information:
|
|
Packit |
961e70 |
Intel Corporation, www.intel.com
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
BSD LICENSE
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Copyright(c) 2017 Intel Corporation.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Redistribution and use in source and binary forms, with or without
|
|
Packit |
961e70 |
modification, are permitted provided that the following conditions
|
|
Packit |
961e70 |
are met:
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
* Redistributions of source code must retain the above copyright
|
|
Packit |
961e70 |
notice, this list of conditions and the following disclaimer.
|
|
Packit |
961e70 |
* Redistributions in binary form must reproduce the above copyright
|
|
Packit |
961e70 |
notice, this list of conditions and the following disclaimer in
|
|
Packit |
961e70 |
the documentation and/or other materials provided with the
|
|
Packit |
961e70 |
distribution.
|
|
Packit |
961e70 |
* Neither the name of Intel Corporation nor the names of its
|
|
Packit |
961e70 |
contributors may be used to endorse or promote products derived
|
|
Packit |
961e70 |
from this software without specific prior written permission.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
Packit |
961e70 |
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
Packit |
961e70 |
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
Packit |
961e70 |
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
Packit |
961e70 |
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
Packit |
961e70 |
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
Packit |
961e70 |
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
Packit |
961e70 |
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
Packit |
961e70 |
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
Packit |
961e70 |
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
Packit |
961e70 |
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Copyright (c) 2003-2017 Intel Corporation. All rights reserved.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
================================================================================
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
ABSTRACT
|
|
Packit |
961e70 |
--------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Discusses how to build, install and test the PSM2 library source code.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Contains the following sections:
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
- INTRODUCTION
|
|
Packit |
961e70 |
- DEPENDENCIES
|
|
Packit |
961e70 |
- BUILDING
|
|
Packit |
961e70 |
* BUILDING USING MAKEFILE
|
|
Packit |
961e70 |
* BUILDING USING RPMBUILD (CREATING SOURCE AND BINARY RPM'S)
|
|
Packit |
961e70 |
- INSTALLING
|
|
Packit |
961e70 |
* INSTALLING USING MAKEFILE
|
|
Packit |
961e70 |
* INSTALLING USING EITHER YUM OR DNF
|
|
Packit Service |
7ed5cc |
- TESTING
|
|
Packit |
961e70 |
- RELATED SOFTWARE TO PSM2
|
|
Packit |
961e70 |
- SUPPORTING DOCUMENTATION
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
INTRODUCTION
|
|
Packit |
961e70 |
============
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This README file discusses how to build, install and test the PSM2 library
|
|
Packit |
961e70 |
source code.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
The PSM2 library supports a number of fabric media and stacks, and all of
|
|
Packit |
961e70 |
them run on version 7.X of Red Hat Enterprise Linux (abbreviated: RHEL), and
|
|
Packit |
961e70 |
SuSE SLES.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Only the x86_64 architecture is supported.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Building PSM2 is possible on RHEL 7.2+ as it ships with hfi1 kernel driver.
|
|
Packit |
961e70 |
For older RHEL 7.x versions and SuSE SLES, OPA is not natively supported
|
|
Packit |
961e70 |
in the kernel and therefore, building PSM2 is not possible unless
|
|
Packit |
961e70 |
you have the correct kernel-devel package or use latest versions of IFS.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
There are two mechanisms for building and installing the PSM2 library:
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
1. Use provided Makefiles to build and install or
|
|
Packit |
961e70 |
2. Generate the *.rpm files which you can then install using either
|
|
Packit |
961e70 |
yum or dnf command
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
DEPENDENCIES
|
|
Packit |
961e70 |
============
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
The following packages are required to build the PSM2 library source code:
|
|
Packit |
961e70 |
(all packages are for the x86_64 architecture)
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
compat-rdma-devel
|
|
Packit |
961e70 |
gcc-4.8.2
|
|
Packit |
961e70 |
glibc-devel
|
|
Packit |
961e70 |
glibc-headers
|
|
Packit |
961e70 |
kernel-headers
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Additional packages for GPU Direct support include:
|
|
Packit |
961e70 |
NVIDIA CUDA toolkit 8.0 or greater. Older versions are not supported.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
In addition to depending on these packages, root privileges are required to
|
|
Packit |
961e70 |
install the runtime libraries and development header files into standard
|
|
Packit |
961e70 |
system location.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
BUILDING
|
|
Packit |
961e70 |
========
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
The instructions below use $BASENAME, $PRODUCT and $RELEASE to refer to
|
|
Packit |
961e70 |
the base name of the tarball, RPM that will be generated and the product
|
|
Packit |
961e70 |
and release identifiers of the RPM.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
The base name of the RPM changes depending on which version/branch
|
|
Packit |
961e70 |
of code you derive the tar file from.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Up until v10.2 of PSM2, the base name for the RPM is hfi1-psm.
|
|
Packit |
961e70 |
From v10.2 onwards, the base name will be libpsm2. The internal
|
|
Packit |
961e70 |
library remains unchanged and is still libpsm2.so.2.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
BUILDING USING MAKEFILES
|
|
Packit |
961e70 |
------------------------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
1. Untar the tarball:
|
|
Packit |
961e70 |
$ tar zxvf $BASENAME-$PRODUCT-$RELEASE.tar.gz
|
|
Packit |
961e70 |
2. Change directory into the untarred location:
|
|
Packit |
961e70 |
$ cd $BASENAME-$PRODUCT-$RELEASE
|
|
Packit |
961e70 |
3. Build:
|
|
Packit |
961e70 |
3.1. To build with GNU C (gcc), run make on the command line:
|
|
Packit |
961e70 |
$ make
|
|
Packit |
961e70 |
- or -
|
|
Packit |
961e70 |
$ make CCARCH=gcc
|
|
Packit |
961e70 |
3.2 To build with Intel C (icc), specify the correct CCARCH:
|
|
Packit |
961e70 |
$ make CCARCH=icc
|
|
Packit |
961e70 |
3.3. To build with CUDA support, specify PSM_CUDA=1
|
|
Packit |
961e70 |
on the command line along with the desired compiler:
|
|
Packit |
961e70 |
$ make PSM_CUDA=1 CCARCH=gcc
|
|
Packit |
961e70 |
- or -
|
|
Packit |
961e70 |
$ make PSM_CUDA=1 CCARCH=icc
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
BUILDING USING RPMBUILD
|
|
Packit |
961e70 |
-----------------------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
1. Run this command from your $PWD to generate rpm, srpm files
|
|
Packit |
961e70 |
$ ./makesrpm.sh a
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This command results in the following collection of rpm's and source
|
|
Packit |
961e70 |
code rpm's under your $PWD/temp.X/ directory.
|
|
Packit |
961e70 |
("X" is the pid of the bash script that created the srpm and rpm files)
|
|
Packit |
961e70 |
(Result shown here for RHEL systems.)
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-compat-10.3.7-1x86_64.rpm
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-devel-10.3.7-1x86_64.rpm
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-10.3.7-1x86_64.rpm
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-debuginfo-10.3.7-1x86_64.rpm
|
|
Packit |
961e70 |
SRPMS/libpsm2-10.3.7-1.src.rpm
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
1.1. Optionally for GPU Direct support run this command from your $PWD to
|
|
Packit |
961e70 |
generate rpm, srpm files
|
|
Packit |
961e70 |
$ ./makesrpm.sh a -cuda
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This command results in the following collection of rpm's and source code
|
|
Packit |
961e70 |
rpm's under your $PWD/temp.X/ directory. ("X" is the pid of the bash
|
|
Packit |
961e70 |
script that created the srpm and rpm files):
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-10.3.7-1cuda.x86_64.rpm
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-compat-10.3.7-1cuda.x86_64.rpm
|
|
Packit |
961e70 |
RPMS/x86_64/libpsm2-devel-10.3.7-1cuda.x86_64.rpm
|
|
Packit |
961e70 |
SRPMS/x86_64/libpsm2-10.3.7-1cuda.src.rpm
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
On systems with SLES 12.3 or newer, the package name for the base libpsm2
|
|
Packit |
961e70 |
RPM will be:
|
|
Packit |
961e70 |
libpsm2-2-10.3.7-1.x86_64.rpm
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Other supporting RPM package names will be as listed above.
|
|
Packit |
961e70 |
|
|
Packit Service |
7ed5cc |
2. To build rpm files from the srpm file with Intel C (icc), specify the
|
|
Packit Service |
7ed5cc |
correct CCARCH in the rpmbuild environment:
|
|
Packit Service |
7ed5cc |
$ env CCARCH=icc rpmbuild --rebuild SRPMS/libpsm2-10.3.7-1.src.rpm
|
|
Packit Service |
7ed5cc |
|
|
Packit |
961e70 |
INSTALLING
|
|
Packit |
961e70 |
==========
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
INSTALLING USING MAKEFILE
|
|
Packit |
961e70 |
-------------------------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Install the libraries and header files on the system (as root):
|
|
Packit |
961e70 |
$ make install
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
The libraries will be installed in /usr/lib64, and the header files will
|
|
Packit |
961e70 |
be installed in /usr/include.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
This behavior can be altered by using the "DESTDIR" and "LIBDIR" variables on
|
|
Packit |
961e70 |
the "make install" command line. "DESTDIR" will add a leading path component
|
|
Packit |
961e70 |
to the overall install path and "LIBDIR" will change the path where libraries
|
|
Packit |
961e70 |
will be installed. For example, "make DESTDIR=/tmp/psm-install install" will
|
|
Packit |
961e70 |
install all files (libraries and headers) into "/tmp/psm-install/usr/...",
|
|
Packit |
961e70 |
"make DESTDIR=/tmp/psm-install LIBDIR=/libraries install" will install the
|
|
Packit |
961e70 |
libraries in "/tmp/psm-install/libraries" and the headers in
|
|
Packit |
961e70 |
"/tmp/psm-install/usr/include", and "make LIBDIR=/tmp/libs install" will
|
|
Packit |
961e70 |
install the libraries in "/tmp/libs" and the headers in "/usr/include".
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
INSTALLING USING EITHER YUM OR DNF
|
|
Packit |
961e70 |
----------------------------------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
You can install the rpm's and source rpm's previously built using rpmbuild using
|
|
Packit |
961e70 |
either the yum or dnf command as the root user. See the appropriate man page for
|
|
Packit |
961e70 |
details of installing rpm's.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Note: It is also possible to use rpm command to install rpm's, but it is recommended
|
|
Packit |
961e70 |
that one use yum/dnf as rpm tool has issues with name changes and obsoletes tags.
|
|
Packit |
961e70 |
yum or dnf should be better able to resolve dependency issues.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
RELATED SOFTWARE TO PSM2
|
|
Packit |
961e70 |
========================
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
MPI Libraries supported
|
|
Packit |
961e70 |
-----------------------
|
|
Packit |
961e70 |
A large number of open source (Open MPI, MVAPICH2) and Vendor MPI
|
|
Packit |
961e70 |
implementations support PSM2 for optimized communication on HCAs. Vendor MPI
|
|
Packit |
961e70 |
implementations (HP-MPI, Intel MPI 4.0 with PMI, Platform/Scali MPI)
|
|
Packit |
961e70 |
require that the PSM2 runtime libraries be installed and available on
|
|
Packit |
961e70 |
each node. Usually a configuration file or a command line switch to mpirun
|
|
Packit |
961e70 |
needs to be specified to utilize the PSM2 transport.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Open MPI support
|
|
Packit |
961e70 |
---------------
|
|
Packit |
961e70 |
If using a version of Open MPI that is not packaged within IFS release, it
|
|
Packit |
961e70 |
is required to use at least v1.10.4. Older versions are not supported. Since
|
|
Packit |
961e70 |
v1.10.4 is not in active development, it is further recommended to use upstream
|
|
Packit |
961e70 |
versions v2.1.2 or newer.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
If NVIDIA* CUDA* support is desired, you can use Open MPI built with CUDA*
|
|
Packit |
961e70 |
support provided by Intel in the IFS installer 10.4 or newer. This Open MPI
|
|
Packit |
961e70 |
build is identified with the "-cuda-hfi" tag to the Open MPI base version
|
|
Packit |
961e70 |
name. The NVIDIA* CUDA* support changes have also been accepted into v2.1.3,
|
|
Packit |
961e70 |
v3.0.1 and v3.1.0 branches of upstream Open MPI repository.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
PSM2 header and runtime files need to be installed on a node where the Open MPI
|
|
Packit |
961e70 |
build is performed. All compute nodes additionally should have the PSM2 runtime
|
|
Packit |
961e70 |
libraries available on them. Open MPI provides a standard configure, make and
|
|
Packit |
961e70 |
make install mechanism which will detect and build the relevant PSM2 network
|
|
Packit |
961e70 |
modules for Open MPI once the header and runtime files are detected.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
MVAPICH2 support
|
|
Packit |
961e70 |
----------------
|
|
Packit |
961e70 |
MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware.
|
|
Packit |
961e70 |
OPA IFS supports MVAPICH2 v2.1 (or later). PSM2 header and runtime files
|
|
Packit |
961e70 |
need to be installed on a node where MVAPICH2 builds are performed. All
|
|
Packit |
961e70 |
compute nodes should also have the PSM2 runtime libraries available on them.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
For building and installing MVAPICH2 with OPA support, refer to MVAPICH2
|
|
Packit |
961e70 |
user guides here:
|
|
Packit |
961e70 |
http://mvapich.cse.ohio-state.edu/userguide/
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
(Note: Support for PSM2 is included in v2.2 and newer)
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
OFED Support
|
|
Packit |
961e70 |
------------
|
|
Packit |
961e70 |
Intel OPA is not yet included within OFED. But the hfi1 driver is available
|
|
Packit |
961e70 |
publicly at kernel.org.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
SUPPORTING DOCUMENTATION
|
|
Packit |
961e70 |
------------------------
|
|
Packit |
961e70 |
PSM2 Programmer's Guide is published along with documentation for "IntelĀ® Omni-Path
|
|
Packit |
961e70 |
Host Fabric Interface PCIe Adapter 100 Series"
|
|
Packit |
961e70 |
(https://www.intel.com/content/www/us/en/support/articles/000016242/network-and-i-o/fabric-products.html)
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Refer to this document for description on APIs and environment variables that
|
|
Packit |
961e70 |
are available for use. For sample code on writing applications leveraging the
|
|
Packit |
961e70 |
PSM2 APIs, refer to Section 5.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
PSM Compatibility Support
|
|
Packit |
961e70 |
------------
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
libpsm2-compat suppports applications that use the PSM API instead of
|
|
Packit |
961e70 |
the PSM2 API, through a compatibility library. This library is an interface
|
|
Packit |
961e70 |
between PSM applications and the PSM2 API.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
If the system has an application that is coded to use PSM and has requirements
|
|
Packit |
961e70 |
to use PSM2 (i.e. the host has Omni-Path hardware), the compatibility library
|
|
Packit |
961e70 |
must be used.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Please refer to your operating system's documentation to find how to modify the
|
|
Packit |
961e70 |
order in which system directories are searched for dynamic libraries. The
|
|
Packit |
961e70 |
libpsm2-compat version of libpsm_infinipath.so.1 must be earlier on the search
|
|
Packit |
961e70 |
path than that of libpsm-infinipath. Doing so allows applications coded to PSM
|
|
Packit |
961e70 |
to transparently use the PSM2 API and devices which require it.
|
|
Packit |
961e70 |
|
|
Packit |
961e70 |
Please note that the installation path for the libpsm2-compat version of
|
|
Packit |
961e70 |
libpsm_infinipath.so.1 will differ depending on your operating system
|
|
Packit |
961e70 |
specifics. Common locations include:
|
|
Packit |
961e70 |
- /usr/lib64/psm2-compat/
|
|
Packit |
961e70 |
- /usr/lib/psm2-compat/
|
|
Packit |
961e70 |
|