Copyright (c) 2015, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL, THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. EXPORT LAWS: THIS LICENSE ADDS NO RESTRICTIONS TO THE EXPORT LAWS OF YOUR JURISDICTION. It is licensee's responsibility to comply with any export regulations applicable in licensee's jurisdiction. Under CURRENT (May 2000) U.S. export regulations this software is eligible for export from the U.S. and can be downloaded by or otherwise exported or reexported worldwide EXCEPT to U.S. embargoed destinations which include Cuba, Iraq, Libya, North Korea, Iran, Syria, Sudan, Afghanistan and any other country to which the U.S. has embargoed goods and services. [ICS VERSION STRING: unknown] README for Lsf scripts directory -------------------------------- This directory contains: * Four scripts for submitting LSF jobs, one each for OpenMPI, mvapich, and mvapich2, plus a fourth for using standard OpenMPI (verbs), which is tightly integrated with LSF * A substitute script for ssh that is required for using MPI with LSF, as per LSF documentation * A sample of the edited openmpi_wrapper from LSF code * A vi input script to modify openmpi_wrapper to work with IntelOPA OFED OpenMPI verbs * This file Virtual Fabric integration with LSF ----------------------------------- The sole intent of the LSF scripts is to act as templates for the administrator, on how to enable LSF to take advantage of Virtual Fabric (vFabric) functionality. In order to make these scripts accessible to common users, the administrator must do the following: 1. (Optional) If necessary, edit the scripts to meet the configuration and usage criteria requirements specific to your LSF environment. 2. Copy the scripts to a shared file system service that provides the appropriate access rights for common users. In order for LSF to take advantage of vFabric functionality, vFabric must be integrated with LSF. There are several integration methods that could be used, these scripts use the method of LSF based queues. This method requires that the administrator perform the following steps (for detailed information on vFabric, reference "Fabric Manager Users Guide"): 1. For LSF, create a queue for each vFabric defined within the FM configuration file (/etc/sysconfig/ifs_fm.xml). 2. Configure each vFabric queue defined within LSF with the compute nodes assigned to it within the FM configuration file. 3. (Optional) Configure each vFabric queue defined within LSF with the priorities, policies, QoS, and attributes that are specific to the needs of the organization and fabric environment. Configuring Nodes for LSF ------------------------- It is recommended that the changes described in this README be performed on or propogated to all nodes in the fabric. This will ensure proper job submission and operation. Fast Fabric may be used to set up password-less ssh. Once that is set up, it can be used to copy the changed files to all nodes in the fabric. LSF Submit Scripts ------------------ LSF submit scripts utilize the LSF primitive "bsub", encapsulating the arguments and call structure for ease-of-use. They query the SA for information regarding virtual fabrics, or in the case of PSM with dist_sa, pass the appropriate arguments to PSM for SA lookup. The submit scripts default the MPICH_PREFIX if it is not set in the environment, although it is recommended to be set explicitly in the environment to avoid any confusion. The mvapich and mvapich2 script default to the -qlc versions of MPI. There are two openmpi scripts; one for standard mpi and one for the -qlc version. The bsub is different for standard openmpi, so it is in its own script. Here are some examples of how to call the scripts and pass vFabric parameters: Example 1: lsf.submit.openmpi-q.job -p 2 -V vf_name -n Compute -d 3 osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2, and the virtual fabric to use is identified by name, and that name is "Compute". Lastly, option 3 is given to denote a dispersive routing option of static_dest. Example 2: lsf.submit.openmpi-q.job -p 2 -V sid -s 0x0000000000000022 -d 1 osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2, and the virtual fabric to use is identified by service id, and that id is "0x0000000000000022". Lastly, option 1 is given to denote a dispersive routing option of adaptive. Example 3: lsf.submit.openmpi-q.job -p 2 -V sid_qlc -s 0x1000117500000000 -q night osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2. The virtual fabric to use is identified by service id, and that service id is "0x1000117500000000". The job will be passed to the QIB driver which will perform the vFabric lookup on behalf of the job. No dispersive routing instructions are given. The job is being requested for submission via the "night" LSF queue. To view the vFabric parameters associated with the job, use the "bjobs -l " command in LSF for the long format of output. LSF Script Administration ------------------------- The scripts are intended to be tailored by the LSF administrator in order to structure the bsub parameters appropriately. Any parameters, such as queue name, application profile, etc. may be inserted into the script either using script options, accessing environment variables, or by hardcoding. Additionally, the scripts use mpirun_rsh to start MPI jobs. This method enables the vFabric parameters to be passed to the MPI jobs so that the appropriate lower level actions can be taken based on those settings (e.g. inserting Service Level information into data messages). Any MPI job command parameters should be used on the script command line, and will be passed to the MPI run job. SSH Script ---------- In accordance with instructions in the Platform document "Integrating LSF's blaunch with MPI Applications", a script for ssh is provided, so that LSF may intercept MPI jobs and convey them to the blaunch facility. The script is constucted in such a way so that normal users of ssh are not affected, while LSF usage is bypassed to blaunch. Instuctions to install the ssh script: 1. Log in as root 2. cd /usr/bin 3. mv ssh ssh.bin 4. copy ssh.script as /usr/bin/ssh Modifying openmpi_wrapper ------------------------- The openmpi_wrapper script provided with LSF, located in LSF_BINDIR, contains a hard-coded path to the OpenMPI directory as installed by Platform's HPC kit. Since this installation supercedes that one with the Intel kit, the location of the OpenMPI files is different. Therefore, the openmpi_wrapper script needs to be modified. The ex script, edit.openmpi.wrapper, may be used to modify the openmpi_wrapper. It uses an ex editor session to change the path from $MPIHOME to $MPICH_PREFIX. The file may be modified by using the script while logged in as root, as follows: ex /opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/openmpi_wrapper < edit.openmpi.wrapper If editing with the script fails, openmpi_wrapper may be edited manually to ensure that the path to the MPIRUN_CMD is correct. Configuration changes in lsf.conf --------------------------------- The following change should be made to the lsf.conf file in order to ensure proper operation: - add "LSF_ROOT_REX=all" The LSF daemons need to be restarted for the configuration change to take effect.