Intel MPI

Set-up on Owens

To configure your environment for the default version of Intel MPI, use module load intelmpi. To configure your environment for a specific version of Intel MPI, use module load intelmpi/<version>. For example, use module load intelmpi/2019.7 to load Intel MPI version 2019.7 on Owens.

You can use module spider intelmpi to view available modules on Owens.

Using Intel MPI

Software compiled against this module will use the libraries at runtime.

Building With Intel MPI

On Ruby, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

VARIABLE	USE
$MPI_CFLAGS	Use during your compilation step for C programs.
$MPI_CXXFLAGS	Use during your compilation step for C++ programs.
$MPI_FFLAGS	Use during your compilation step for Fortran programs.
$MPI_F90FLAGS	Use during your compilation step for Fortran 90 programs.
$MPI_LIBS	Use when linking your program to Intel MPI.

In general, for any application already set up to use mpicc, (or similar), compilation should be fairly straightforward.

Batch Usage on Owens

When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

Non-interactive Batch Job (Parallel Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Owens:

#!/bin/bash
#SBATCH --job-name MyIntelMPIJob
#SBATCH --nodes=4 --ntasks-per-node=28
#SBATCH --time=5:00:00
#SBATCH --account=<project-account>

module load intelmpi
srun my-impi-application

Set-up on Pitzer

To configure your environment for the default version of Intel MPI, use module load intelmpi.

Using Intel MPI

Software compiled against this module will use the libraries at runtime.

Building With Intel MPI

On Oakley, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

VARIABLE	USE
$MPI_CFLAGS	Use during your compilation step for C programs.
$MPI_CXXFLAGS	Use during your compilation step for C++ programs.
$MPI_FFLAGS	Use during your compilation step for Fortran programs.
$MPI_F90FLAGS	Use during your compilation step for Fortran 90 programs.
$MPI_LIBS	Use when linking your program to Intel MPI.

In general, for any application already set up to use mpicc compilation should be fairly straightforward.

Batch Usage on Pitzer

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

Non-interactive Batch Job (Parallel Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Pitzer:

#!/bin/bash
#SBATCH --job-name MyIntelMPIJob
#SBATCH --nodes=2 --ntasks-per-node=48
#SBATCH --time=5:00:00
#SBATCH --account=<project-account>

module load intelmpi
srun my-impi-application

[mpiexec@o0439.ten.osc.edu] wait_proxies_to_terminate (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:532): downstream from host o0439 was killed by signal 11 (Segmentation fault)
[mpiexec@o0439.ten.osc.edu] main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:2114): assert (exitcodes != NULL) failed

/var/spool/torque/mom_priv/jobs/11510761.owens-batch.ten.osc.edu.SC: line 30: 11728 Segmentation fault  

/var/spool/slurmd/job00884/slurm_script: line 24:  3180 Segmentation fault      (core dumped)

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=N

unset I_MPI_PMI_LIBRARY 
export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0   # the option -ppn only works if you set this before