Supercomputing Networking Research Education Ohio Supercomputer Center Site Map Staff Directory Support
Welder image

bcMPI - utilities

autobuild-bcmpi.sh

description

Shell script to bootstrap the checkout and build process using the developer recommended conventions. Currently requires subversion access, non-developers may still find it useful as a reference.

usage

Download and save to your $HOME directory.

Create a working directory and cd into it:

% mkdir -p /tmp/${USER}/bcMPI
% cd /tmp/${USER}/bcMPI

Checkout and build with chosen configuration:

% sh ${HOME}/autobuild-bcmpi.sh kodos-openmpi-ib-oct64mpi

bcmpi_test

description

Installed in bcmpi/regress starting with bcMPI-1.1.

A standalone C program which tests the target MPI environment configuration without complications from the MATLAB or Octave interpreter. Successful completion indicates that programs can be started, the MPI and bcMPI libraries can be found, and basic MPI communication is functioning.

usage

Create and submit or run the bcmpi_test.pbs script:

% qsub bcmpi_test.pbs
-or-
% sh bcmpi_test.pbs

      

output

Successful output will contain a bunch of lines ending with "err=0". The final lines should be "bcmpi_finalize(): err=0". Any failure will require fixing your environment configuration before the bcMPI toolboxes can be expected to work.


create_job_file.py

description

A python script to generate batch system job files. It can create PBS, LSF or SH job files. Information to create a job file is read from installation, queue, job specific configuration files and command line options.

Input values to this script may be specified in configuration files or as command line options or both. Command line option values overwrite configuration file values.

configuration options

--help
Shows help message
--config-file
Name of configuration file(s). More than one configuration file may be specified.
--interpreter
Name of interpreter for an installation which is specified in the configuration file. Valid inteprepter names are "octave" or "matlab". "None" may used to create SH scripts.
--batch-system
Name of batch system used to run the job. Valid batch system names are "PBS", "LSF" and "SH". When "SH" is specified, the batch shell variable is used to generate either a .sh or .csh file
--batch-shell
Name of batch shell used by a specific batch system. PBS uses "sh" and LSF uses "csh".
--mpi-library
Name of mpi library used for an installtion which is specified in the configuration file.
--pernode
Use one processor per node. Set to "True" to run in a non-smp machine and to "False" for a smp machine.
--procs-pernode
Set this to the number of processors per node for a smp machine or to "1" for a non-smp machine. Number of nodes requested for a job in a smp machine is calculated using values in "procs-pernode" and "processes" options.
--processes
Total number of processes to be started for a submitted job.
--queue
Batch system destination queue.
--machine-name
Name of the machine supported by the job generation script. It is used to select the job file header for the batch system used in that machine.
--account
Account associated with the job. Used for generating LSF scripts and ignored for PBS scripts.
--walltime
Maximum amount of time to run the job.
--programs
A comma sepearte list of matlab program file(s) to run.
--job-name
Job name to the batch system. If not specified, base name of the first program file is used.
--batch-script
File name for the generator script to write the output. If not specified, stdout is used.
--join-output
When set to "True", merges stdout and stderr (deafult)

configuration files

ParaM.pjc - The installation specific variables are set and installed in etc/ParaM.pjc file. Read by the job file generation script by default.

queue.pjc - Batch system target queue specific variables can be set in queue.pjc file. An example is installed in share/examples. User has to copy this file, modify it (optional) and specify it in the command line.

Job specific variables may be set in a user defined configuration file. This file follows the same format as queue configuration file.

usage

To generate a job script example.pbs for program file example.m, using user defined queue.pjc:

% create_job_file.py -c queue.pjc -o example.pbs -p example.m

      

To create a job script using user defined queue.pjc and job specifc variables in config.job:

% create_job_file.py -c queue.pjc -c config.job

      

output

Creates a batch system job file specified in batch-script option or redirects the output to stdout.


create_job_files.sh

A warpper script for job creation script that creates batch system job files for the program files in regress and examples directories. User may modify the m files list and options to the job creation script.

usage

If queue cofiguration file is not specified in the command line, uses example config file.

% ./create_job_files.sh 
        
      

Users may copy and modify example target queue config file in share/examples/queue.pjc.

% ./create_job_files.sh queue.pjc
        
      

output

Creates a batch system job file specified in batch-script option or redirects the output to stdout.


environment setup

description

The environ files are responsible for setting all the environment variables required for running bcMPI applications. The appropriate file should be sourced by the shell script or at the command prompt.

usage

The non-interpreter specific variables are set in environ.csh and environ.sh . It may be necessary to edit these files after installation to work with your site's environment (eg, loading modules at OSC).

Users should source one of the interpreter specific environ files, these will in turn source the common files:

For csh shells:

% source octave-environ.csh
-or-
% source matlab-environ.csh
	
      

For sh shells:

$ . octave-environ.sh
-or-
$ . matlab-environ.sh
	
      

mpi_matlab

description

A wrapper script that enables the MATLAB interpreter to be started in an MPI environment. It starts by moving the command line parameters passed by mpiexec (or mpirun) to the environment. It next builds a new command line to start the interpreter, and creates a temporary standard input file with instructions to addpath() the bcMPI toolbox and declare MPI_COMM_WORLD (for compatibility with MatlabMPI). Finally, it execs the interpreter.

It also contains preliminary code to monitor the running MATLAB process. This feature was never finished because the PBS job scheduler was found to be killing the mpi_matlab script but leaving MATLAB running, consuming resources and eventually preventing additional jobs from being run. Code remains for developer diagnostics.

interpreter options

MATLAB
The name of the matlab interpreter executable. Value is built using configuration options, should not need to be changed.
MATLAB_FLAGS
Flags passed on the interpreter command line to make it play nicely in the MPI environment. Can be changed in your configuration
MATLAB_COMMAND
The top level command to be run by the interpreter. Typically is the name of a script (without the .m) or function to call. Set in the environment if your mpirun or mpiexec can propogate variables, otherwise create a copy of mpi_octave and edit.

MPI flags

Set the environment variable MPI_MATLAB_FLAGS to control the behaviour of mpi_matlab. Note that the default value used by installed tests and examples can be changed in your configuration.
-stdout <filename>
Redirect stdout to a file, using substitutions below.
-stderr <filename>
Redirect stderr to a file, using substitutions below.
-std-files
Sets stdout and stderr to '%(jobname)s-%(rank)02i.%(type)s%(jobid)i'
-oe
Redirects stderr to stdout.

Filename substitutions

The filename for each MPI process must be unique, otherwise all processes will open the same file and most output will be lost. Note that not all substitutions are available for all MPI environments, experiment to see what works best.

The default is to not redirect stdout or stderr, the final behaviour is then determined by the MPI environment.

filename substitution values

%(command)s The base value of MATLAB_COMMAND .
%(jobid)i The PBS supplied value in PBS_JOBID , or the user supplied value in MPI_MATLAB_JOBID , otherwise 0.
%(jobname)s The PBS supplied value in PBS_JOBNAME , or the user supplied value in MPI_MATLAB_JOBNAME , otherwise %(command)s .
%(rank)i The MPI rank, when available (OpenMPI), otherwise %(pid)i .
%(pid)i The process ID of the interpreter.
%(type)s "o" if used with -stdout, "e" if used with -stderr.

mpi_octave

description

Same as mpi_matlab, except it works with Octave.

interpreter options

OCTAVE
The name of the Octave interpreter executable. Value is built using configuration options, should not need to be changed.
OCTAVE_FLAGS
Flags passed on the interpreter command line to make it play nicely in the MPI environment. Can be changed in your configuration.
OCTAVE_COMMAND
The top level command to be run by the interpreter. Typically is the name of a script (without the .m) or function to call. Set in the environment if your mpirun or mpiexec can propogate the value, otherwise create a copy of mpi_octave and edit.

MPI flags

Set the environment variable MPI_OCTAVE_FLAGS to control the behaviour of mpi_octave. Supported options are the same as for mpi_matlab. Note that the default value used by installed tests and examples can be changed in your configuration.


mpi_setup

The environment provided by MPI is not guaranteed until after MPI_Init() has been called. This is problematic for bcMPI because the interpreter must be started before MPI_Init() can be called. Use mpi_setup to generate a portable shell script which will initialize paths, change to the correct working directory, and set the command to be run. After initialization, the generated script will exec mpi_matlab or mpi_octave.

usage examples

To run example.m using MATLAB:

% mpi_setup example matlab MATLAB_COMMAND=example
% mpirun example
      

By convention, the name of the generated script (first parameter to mpi_setup) is the name of the matlab script without the ".m". The second parameter is the name of the interpreter ("matlab" or "octave").

Either MPI_MATLAB_FLAGS and MPI_OCTAVE_FLAGS will always be propogated:

% setenv MPI_MATLAB_FLAGS -oe
% mpi_setup example matlab MATLAB_COMMAND=example
% mpirun example
      

Additional environment variables may be set by adding name=value to the mpi_setup command:

% mpi_setup example octave \
	OCTAVE_COMMAND=example \
	FOO=bar
% mpirun example
      

Empty values are supported, but currently there is no way to force a variable to be undefined.

Parameters to the mpi wrapper may be adedd after variables, although this is intended for developer use only:

% mpi_setup example octave \
	OCTAVE_COMMAND=example \
	FOO=bar \
	--qwerty
% mpirun example
      

mpistart.py

description

Unsupported developer debugging tool. Replacement for MPICH's mpirun, starts all processes on the local host for quick debugging. Must be manually installed.