Copyright (c) 2015, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL, THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. EXPORT LAWS: THIS LICENSE ADDS NO RESTRICTIONS TO THE EXPORT LAWS OF YOUR JURISDICTION. It is licensee's responsibility to comply with any export regulations applicable in licensee's jurisdiction. Under CURRENT (May 2000) U.S. export regulations this software is eligible for export from the U.S. and can be downloaded by or otherwise exported or reexported worldwide EXCEPT to U.S. embargoed destinations which include Cuba, Iraq, Libya, North Korea, Iran, Syria, Sudan, Afghanistan and any other country to which the U.S. has embargoed goods and services. [ICS VERSION STRING: unknown] README for Moab scripts directory --------------------------------- This directory contains: * Three scripts for submitting Moab jobs, one each for OpenMPI, mvapich, and mvapich2 * A helper script that is used by the three previously mentioned scripts * This file Virtual Fabric integration with Moab ------------------------------------ The sole intent of the Moab scripts is to act as templates for the administrator, on how to enable Moab to take advantage of Virtual Fabric (vFabric) functionality. In order to make these scripts accessible to common users, the administrator must do the following: 1. (Optional) If necessary, edit the scripts to meet the configuration and usage criteria requirements specific to your Moab environment. 2. Copy the scripts to a shared file system service that provides the appropriate access rights for common users. In order for Moab to take advantage of vFabric functionality, vFabric must be integrated with Moab. There are several integration methods that could be used, these scripts use the method of Moab based queues. This method requires that the administrator perform the following steps (for detailed information on vFabric, reference "Fabric Manager Users Guide"): 1. For both Moab and the Resource Manager(s), create a queue for each vFabric defined within the FM configuration file (/etc/sysconfig/ifs_fm.xml). 2. Configure each vFabric queue defined within Moab with the compute nodes assigned to it within the FM configuration file. 3. (Optional) Configure each vFabric queue defined within Moab with the priorities, policies, QoS, and attributes that are specific to the needs of the organization and fabric environment. Note, depending on the queue support requirements of a Resource Manager, steps 2 and 3 may also have to be performed upon that Resource Manager (TORQUE does not require these two steps). Moab Submit Scripts ------------------- Moab submit scripts utilize the Moab primitive "msub", encapsulating the arguments and call structure for ease-of-use. They query the SA for information regarding virtual fabrics, or in the case of PSM with dist_sa, pass the appropriate arguments to PSM for SA lookup. The submit scripts default the MPICH_PREFIX if it is not set in the environment. The openmpi, mvapich and mvapich2 scripts default to the -qlc versions of MPI. The submit scripts utilize a helper script called "moab.mpi.job.wrapper". This helper script is used to dynamically generate a hosts file required for MPI. This host file is a list of hosts to run MPI on. The host file created is called ".mpi_hosts_file." and is created in the work directory on the primary node executing the job. Note, the Moab scripts DO NOT delete the host file; therefore, clean up of these temporary host files is the responsibility of the user. Here are some examples of how to call the scripts and pass vFabric parameters (Note, these examples utilize the proprietary MPI applications, which are located in the /usr/src/opa/mpi_apps/ directory. These MPI applications were provided specifically for administrative use only): Example 1: moab.submit.openmpi.job -p 2 -V vf_name -n Compute -d 3 osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2, and the virtual fabric to use is identified by name, and that name is "Compute". Option 3 is given to denote a dispersive routing option of static_dest. Example 2: moab.submit.openmpi.job -p 2 -V sid -s 0x0000000000000022 -d 1 osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2, and the virtual fabric to use is identified by service id, and that id is "0x0000000000000022". Option 3 is given to denote a dispersive routing option of adaptive. Example 3: moab.submit.openmpi.job -p 2 -V sid_qlc -s 0x1000117500000000 -q night osu2/osu_bibw This submits a job with the openmpi-qlc subsystem, i.e. the run file osu_bibw is expected to have been compiled with the openmpi-qlc compiler under gcc. The number of processes is given as 2. The virtual fabric to use is identified by service id, and that id is "0x1000117500000000"; it will be passed to the QIB driver which will perform the vFabric lookup on behalf of the job. No dispersive routing instructions are given. The job is being requested for submission via the "night" Moab queue. Moab Script Administration -------------------------- The scripts are intended to be tailored by the Moab administrator in order to structure the msub parameters appropriately. Any parameters, such as queue name, application profile, work directory, etc. may be inserted into the script either using script options, accessing environment variables, or by hardcoding. Additionally, the scripts use mpirun_rsh or mpirun to start MPI jobs. This method enables the vFabric parameters to be passed to the MPI jobs so that the appropriate lower level actions can be taken based on those settings (e.g. inserting Service Level information into data messages). Any MPI job command parameters should be used on the script command line, and will be passed to the MPI run job.