Software List

Ohio Supercomputer Center (OSC) has a variety of software applications to support all aspects of scientific research. We are actively updating this documentation to ensure it matches the state of the supercomputers. This page is currently missing some content; use module spider on each system for a comprehensive list of available software.

Supercomputer: 
Service: 

Abaqus

ABAQUS is a finite element analysis program owned and supported by SIMULIA, the Dassault Systèmes brand for Realistic Simulation.

Availability and Restrictions

Versions

The available programs are ABAQUS/CAE, ABAQUS/Standard and ABAQUS/Explicit. The versions currently available at OSC are:

Version Pitzer Cardinal Notes
2022   X  
2024 X X  
*: Default Version

You can use  module spider abaqus to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

OSC's ABAQUS license can only be used for educational, institutional, instructional, and/or research purposes. Only users who are faculty, research staff, or students at the following institutions are permitted to utilized OSC's license:

  • The Ohio State University
  • University of Toledo
  • University of Cincinnati
  • University of Dayton
  • University of Akron
  • Miami University

Users from additional degree granting academic institutions may request to be added to this list per a cost by contacting OSC Help.

The use of ABAQUS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction. 

(link sends e-mail)

Access for Commercial Users

Contact OSC Help for getting access to ABAQUS if you are a commercial user.

Publisher/Vendor/Repository and License Type

Dassault Systemes, Commercial

Usage

Token Usage

ABAQUS software usage is monitored though a token-based license manager. This means every time you run an ABAQUS job, tokens are checked out from our pool for your usage. To ensure your job is only started when its required ABAQUS tokens are available it is important to include a software flag within your job script's SBATCH directives.  A minimum of 5 tokens are required per job, so a 1 node, 1 processor ABAQUS job would need the following SBATCH software flag:  #SBATCH -L abaqus@osc:5 . Jobs requiring more cores will need to request more tokens as calculated with the formula:  M = int(5 x N^0.422) , where N is the total number of cores.  For common requests, you can refer to the following table:

         Cores            (nodes x cores each):

1 2 3 4 6 8 12 16 28 32 56
Tokens needed: 5 6 7 8 10 12 14 16 20 21 27

Usage on Cardinal

Set-up on Cardinal

To load the default version of ABAQUS, use  module load abaqus . To select a particular software version, use     module load abaqus/version . For example, use  module load abaqus/2022  to load ABAQUS version 2022. 

Using ABAQUS

Example input data files are available with the ABAQUS release. The  abaqus fetch  utility is used to extract these input files for use. For example, to fetch input files for one of the sample problems including 4 input files, type:

abaqus fetch job=knee_bolster 

abaqus fetch job=knee_bolster_ef1 

abaqus fetch job=knee_bolster_ef2 

abaqus fetch job=knee_bolster_ef3 

Also, use the  abaqus help  utility to list all the abaqus execution procedures.

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session on Cardinal, one can run the following command:
sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00 -L abaqus@osc:20
which gives you 28 cores ( -N 1 -n 28 ) for 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

Below is the example batch script ( job.txt ) for a serial run:

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L abaqus@osc:5
#SBATCH --account=<project-account>
#
# The following lines set up the ABAQUS environment
#
module load abaqus
#
cp *.inp $TMPDIR
cd $TMPDIR
#
# Run ABAQUS
#
abaqus job=knee_bolster interactive
#
# Now, copy data (or move) back once the simulation has completed
#
cp * $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the  job.txt  file with the command:  qsub job.txt 

NOTE:

  • Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory ( $SLURM_SUBMIT_DIR ) to  $TMPDIR , and copy your results back at the end of your script. Running your job on  $TMPDIR  ensures maximum efficiency.
  • The keyword  interactive  is required in the execution line  abaqus job=knee_bolster interactive  for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword  interactive  in this case simply tells ABAQUS not to return until the simulation has completed.
  • The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. However, it is better practice to designate the main input file explicitly by adding  input=<my_input_file_name>.inp  to the execution line:  abaqus job=knee_bolster input=<my_input_file_name>.inp interactive .
  • Define  nodes=1  (1<=cores<=48 for Cardinal) for a serial run.
  • If cores > 1, add  cpus=<n>  to the execution line, where n=cores:  abaqus job=test input=<my_input_file_name1>.inp cpus=<n> interactive .
Non-interactive Batch Job (Parallel Run)
Note: abaqus will not run correctly in parallel with input files in $TMPDIR!  Use the scratch file system.

Below is an example batch script ( job.txt ) for a parallel run:

#!/bin/bash 
#SBATCH --time=1:00:00 
#SBATCH --nodes=2 --ntasks-per-node=28 --gres=pfsdir
#SBATCH -L abaqus@osc:27
#SBATCH --account=<project-account>
#
# The following lines set up the ABAQUS environment
#
module load abaqus
#
# Cope input files to /fs/scratch and run Abaqus there
#
cp *.inp $PFSDIR
cd $PFSDIR
#
# Run ABAQUS, note that in this case we have provided the names of the input files explicitly
#
abaqus job=test input=<my_input_file_name1>.inp cpus=$SLURM_NTASKS interactive
#
# Now, move data back once the simulation has completed
#
mv * $SLURM_SUBMIT_DIR

NOTE:

  • If you request a partial node for a serial job (cores<28), you need to add 'mp_mode=threads' option in order to get the full performance.  
  • Specify  cpus=<n>  in the execution line, where n=nodes*cores.
  • Everything else is similar to the serial script above.
  • Usage of a user-defined material (UMAT) script in Fortran is limited on Clusters as follows:
    1. abaqus 2017:  correctly running on single and multi-nodes
    2. abaqus 6.14 and 2016:  correctly running on a single node.

Configuring MPI Environment

  1. Choosing MPI: Create an Abaqus environment file named abaqus_v6.env in the home or working directory, and add one of the following lines to specify the MPI implementation:
    • For IMPI: mp_mpi_implementation = IMPI
    • For PMPI: mp_mpi_implementation = PMPI
  2. Configuring IMPI: If using IMPI, it is necessary to set the correct bootstrap method. Add the environment variable: I_MPI_HYDRA_BOOTSTRAP=ssh

Further Reading

 

Supercomputer: 
Service: 
Fields of Science: 

AFNI

AFNI (Analysis of Functional Neuro Images) is a leading software suite of C, Python, and R programs and shell scripst primarily developed for the analysis and display of multiple MRI modalities: anatomical, functional MRI (FMRI) and diffusion wieghted (DW) data. It is freely available (both as open source code and as precompiled binaries) for research purposes.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

VERSION

Pitzer Ascend Cardinal
2024.10.14   X X*
25.1.15 X X X
* Current default version

You can use module spider afni to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

AFNI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

AFNI is distributed freely under the Gnu General Public License. Major portions of this software were written at the Medical College of Wisconsin, which owns the copyright to that code. For fuller details, see the file http://afni.nimh.nih.gov/pub/dist/src/README.copyright.

Usage

Usage

Set-up

To configure your environment for use of AFNI, run the following command: module load afni. The default version will be loaded. To select a particular AFNI version, use module load afni/version. For example, use module load afni/2021.6.10 to load AFNI 2021.6.10.

AFNI is installed in a container.  AFNI_IMG environment variable contains the container image file path. So, an example usage would be

module load afni
apptainer exec $AFNI_IMG suma

This command will open the SUMA GUI environment, and we recommend Ondemand VDI or Desktop for GUI. 

To launch the AFNI GUI, use the following commands in a terminal:

apptainer shell $AFNI_IMG
afni

This is due to launching AFNI requires two steps, and the first step detaches from the terminal, resulting in a crash if you run afni command directly through the container.

For more information about singularity usages, please read OSC apptainer page

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

AMBER

The Assisted Model Building with Energy Refinement (AMBER) package, which includes AmberTools, contains many molecular simulation programs targeted at biomolecular systems. A wide variety of modelling techniques are available. It generally scales well on modest numbers of processors, and the GPU enabled CUDA programs are very efficient.

Availability and Restrictions

Versions

AMBER is available on the Pitzer and Ascend clusters. The following versions are currently available at OSC (S means serial executables, P means parallel, and C means CUDA, i.e., GPU enabled):

Version Pitzer Ascend Cardinal Notes
24 SPC SPC SPC  
* Current default version
*  IMPORTANT NOTE: You need to load correct compiler and MPI modules before you use Amber. In order to find out what modules you need, use module spider amber/{version} .

You can use module spider amber to view available modules and use module spider amber/{version} to view installation details including applied Amber updates. Feel free to contact OSC Help if you need other versions or executables for your work.

Access for Academic Users

OSC's Amber is available to not-for-profit OSC users; simply contact OSC Help to request the appropriate form for access.

Access for Commercial Users

For-profit OSC users must obtain their own Amber license. 

Publisher/Vendor/Repository and License Type

University of California, San Francisco, Commercial

Usage

Usage on Pitzer

Set-up

To load the default version of AMBER module, use  module load amber

Using AMBER

A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

tleap

Parallel Amber programs must be run in a batch environment with mpiexec, e.g.:

srun pmemd.MPI

 

Batch Usage

When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session, one can run the following command:
sinteractive -A <project-account> -N 1 -n 48 -t 1:00:00
which gives you one node with 48 cores ( -N 1 -n 48) with 1 hour ( -t 1:00:00). You may adjust the numbers per your need.
Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

~srb/workshops/compchem/amber/

Below is the example batch script ( job.txt ) for a serial run:

# AMBER Example Batch Script for the Basic Tutorial in the Amber manual 
#!/bin/bash
#SBATCH --job-name 6pti #
SBATCH --nodes=1 --ntasks-per-node=48 
SBATCH --time=0:20:00
#SBATCH --account=<project-account>

module load amber
# Use TMPDIR for best performance.
cd $TMPDIR
# SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
cp -p $SLURM_SUBMIT_DIR/6pti.prmtop .
cp -p $SLURM_SUBMIT_DIR/6pti.prmcrd .
# Running minimization for BPTI
cat << eof > min.in
# 200 steps of minimization, generalized Born solvent model
&cntrl
maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
/
eof
sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
cp -p min.in 6pti.min1.out 6pti.min1.xyz $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the  job.txt  file with the command:  sbatch job.txt .

Troubleshooting

In general, the scientific method should be applied to usage problems.  Users should check all inputs and examine all outputs for the first signs of trouble.  When one cannot find issues with ones inputs, it is often helpful to ask fellow humans, especially labmates, to review the inputs and outputs.  Reproducibility of molecular dynamics simulations is subject to many caveats.  See page 24 of the Amber18 manual for a discussion.

Further Reading

Supercomputer: 
Service: 

ANSYS

ANSYS offers a comprehensive software suite that spans the entire range of physics, providing access to virtually any field of engineering simulation that a design process requires. Supports are provided by ANSYS, Inc

Availability and Restrictions

Versions

Version Pitzer Cardinal
2024R1 X X

 

OSC has Academic Multiphysics Campus Solution license from Ansys. The license includes most of all the features that Ansys provides. See "Academic Multiphysics Campus Solution Products" in this table for all available products at OSC.

ANSYS only works with versions 2021R1 or newer due to the license upgrade. We are working with the vendor to fix the issue now. 

Access for Academic Users

OSC has an "Academic Research " license for ANSYS. This allows for academic use of the software by Ohio faculty and students, with some restrictions. To view current ANSYS node restrictions, please see ANSYS's Terms of Use.

Use of ANSYS products at OSC for academic purposes requires validation. Please contact OSC Help for further instruction.

Access for Commercial Users

Contact OSC Help for getting access to ANSYS if you are a commercial user.

Publisher/Vendor/Repository and License Type

Ansys, Inc., Commercial

Usage

For more information on how to use each ANSYS product at OSC systems, refer to its documentation page provided at the end of this page.

Known Issues

Simultaneously loading multiple of Fluent and ANSYS module cryptic error

Due to the way our Fluent and ANSYS modules are configured, simultaneously loading multiple of either module will cause a cryptic error. The most common case of this happening is when multiple of a user's jobs are started at the same time and all load the module at once. In order for this error to manifest, the modules have to be loaded at precisely the same time; a rare occurrence, but a probable occurrence over the long term.

If you encounter this error you are not at fault. Please resubmit the failed job(s).

If you frequently submit large amounts of Fluent or ANSYS jobs, we recommend you stagger your job submit times to lower the chances of two jobs starting at the same time, and hence loading the module at the same time. Another solution is to establish job dependencies between jobs, so jobs will only start one after another. To do this, you would add the SLURM directive:

#SBATCH --dependency=after:jobid

To jobs you want to only start after another job has started. You would replace jobid with the job ID of the job to wait for. If you have additional questions, please contact OSC Help.

Ansys DesignModeler with hardware acceleration

Updated: April 2022
Versions Affected:  < 19.1
Ansys DesignModeler with hardware acceleration is not working. With Ansys version greater than 19.1, DesignModeler is working with software rendering mode, but it is very slow.

OMP: System error #22: Invalid argument

If you run into this error:

OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
forrtl: error (76): Abort trap signal

Try setting the environment variable KMP_AFFINITY=disabled before running Ansys.

Further Reading

See Also

Supercomputer: 
Service: 

ANSYS Mechanical

ANSYS Mechanical is a finite element analysis (FEA) tool that enables you to analyze complex product architectures and solve difficult mechanical problems. You can use ANSYS Mechanical to simulate real world behavior of components and sub-systems, and customize it to test design variations quickly and accurately.

Availability and Restrictions

ANSYS Mechanical is available on the Cardinal Cluster. You can see the currently available versions in the table on the main Ansys page here.

You can use module spider ansys to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of ANSYS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

Access for Commercial Users

Contact OSC Help for getting access to ANSYS if you are a commercial user.

Usage

Usage on Cardinal

Set-up on Cardinal

To load the default version of ANSYS module, use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load ANSYS version 17.2. 

Using ANSYS Mechanical

Following a successful loading of the ANSYS module, you can access the ANSYS Mechanical commands and utility programs located in your execution path:

ansys <switch options> <file>

The ANSYS Mechanical command takes a number of Unix-style switches and parameters.

The -j Switch

The command accepts a -j switch. It specifies the "job id," which determines the naming of output files. The default is the name of the input file.

The -d Switch

The command accepts a -d switch. It specifies the device type. The value can be X11, x11, X11C, x11c, or 3D.

The -m Switch

The command accepts a -m switch. It specifies the amount of working storage obtained from the system. The units are megawords.

The memory requirement for the entire execution will be approximately 5300000 words more than the -m specification. This is calculated for you if you use ansnqs to construct an NQS request.

The -b [nolist] Switch

The command accepts a -b switch. It specifies that no user input is expected (batch execution).

The -s [noread] Switch

The command accepts a -s switch. By default, the start-up file is read during an interactive session and not read during batch execution. These defaults may be changed with the -s command line argument. The noread option of the -s argument specifies that the start-up file is not to be read, even during an interactive session. Conversely, the -s argument with the -b batch argument forces the reading of the start-up file during batch execution.

The -g [off] Switch

The command accepts a -g switch. It specifies that the ANSYS graphical user interface started automatically.

ANSYS Mechanical parameters

ANSYS Mechanical parameters may be assigned values on the command. The parameter must be at least two characters long and must be a legal parameter name. The ANSYS Mechanical parameter that is to be assigned a value should be given on the command line with a preceding dash (-), a space immediately after, and the value immediately after the space:

module load ansys
ansys -pval1 -10.2 -EEE .1e6
sets pval1 to -10.2 and EEE to 100000

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ANSYS Mechanical analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info. Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

Interactive mode is similar to running ANSYS Mechanical on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run ANSYS Mechanical interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

To run interactive ANSYS Mechanical, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. For example, the following command requests one core ( -N 1 -n 1  ), for a walltime of 1 hour (  -t 1:00:00 ), with one ANSYS license:

sinteractive -N 1 -n 1 -t 1:00:00 -L ansys@osc:1,ansyspar@osc:24 -A <account>

You may adjust the numbers per your need. This job will queue until resources becomes available. Once the job is started, you're automatically logged in on the compute node; and you can launch ANSYS Mechanical and start the graphic interface with the following commands:

module load ansys
ansys -g
Non-interactive Batch Job (Serial Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. For a given model, prepare the input file with ANSYS Mechanical commands (named  ansys.in  for example) for the batch run. Below is the example batch script (   job.txt ) for a serial run:

#!/bin/bash
#SBATCH --job-name=ansys_test
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L ansys@osc:1
#SBATCH --account=<account>

cd $TMPDIR  
cp $SLURM_SUBMIT_DIR/ansys.in .    
module load ansys  
ansys < ansys.in   
cp <output files> $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the  job.txt  file with the command:  qsub job.txt .

Non-interactive Batch Job (Parallel Run)

To take advantage of the powerful compute resources at OSC, you may choose to run distributed ANSYS Mechanical for large problems. Multiple nodes and cores can be requested to accelerate the solution time. Note that you'll need to change your batch script slightly for distributed runs.

Starting from September 15, 2015, a job using HPC tokens (with "ansyspar" flag) should be submitted to Cardinal clusters due to scheduler issue.

For distributed ANSYS Mechanical jobs, the number of processors needs to be specified in the command line with options '-dis -np':

#!/bin/bash
#SBATCH --job-name=ansys_test
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=28
#SBATCH --account=<account>
#SBATCH -L ansys@osc:1,ansyspar@osc:24

...
ansys -b -dis -mpi ibmmpi -np ${SLURM_NTASKS} -i ansys.in
 
...

Notice that in the script above, the ansys parallel license is requested as well as ansys license in the format of

#SBATCH -L ansys@osc:1,ansyspar@osc:n

where n=m-4, with m being the total cpus called for this job. This part is necessary when the total cpus called is greater than 4 (m>4), which applies for the parallel example below as well.

The following shows changes in the batch script if 2 nodes on Cardinal are requested for a parallel ANSYS Mechanical job:

#!/bin/bash
#SBATCH --job-name=ansys_test
#SBATCH --time=3:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L ansys@osc:1,ansyspar@osc:52

...
ansys -b -dis -mpi ibmmpi -np ${SLURM_NTASKS} -i ansys.in 
...
pbsdcp -g '<output files>' $SLURM_SUBMIT_DIR

The 'pbsdcp -g' command in the last line in the script above makes sure that all result files generated by different compute nodes are copied back to the work directory.

Further Reading

See Also

Supercomputer: 
Service: 

CFX

ANSYS CFX (called CFX hereafter) is a computational fluid dynamics (CFD) program for modeling fluid flow and heat transfer in a variety of applications.

Availability and Restrictions

CFX is available on the Cardinal Cluster. You can see the currently available versions in the table on the main Ansys page here.

You can use module spider ansys  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

Currently, there are in total 50 ANSYS base license tokens and 900 HPC tokens for academic users. These base tokens and HPC tokens are shared with all ANSYS products we have at OSC. A base license token will allow CFX to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial CFX job with 1 core will need 1 base license token while a parallel CFX job with 28 cores will need 1 base license token and 24 HPC tokens.

Access for Commercial Users

Contact OSC Help for getting access to CFX if you are a commercial user.

Usage

Usage on Cardianl

Set-up on Cardinal

To load the default version, use  module load ansys  . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load CFX version 17.2 on Cardinal. 

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

Interactive mode is similar to running CFX on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run CFX interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

To run interactive CFX GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please follwoing the steps below to use CFX GUI interactivly:

  1. Ensure that your SSH client software has X11 forwarding enabled
  2. Connect to Cardinal system
  3. Request an interactive job. The command below will request one core (  -N 1 -n 1  ), for a walltime of one hour ( -t 1:00:00 ), with one ANSYS CFD license (modify as per your own needs):
    sinteractive -N 1 -n 1 -t 1:00:00 -L ansys@osc:1
  4. Once the interactive job has started, run the following commands to setup and start the CFX GUI:

    module load ansys
    cfx5 
    
Non-interactive Batch Job (Serial Run Using 1 Base Token)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

Below is the example batch script (  job.txt ) for a serial run with an input file test.def ) :

#!/bin/bash
#SBATCH --job-name=serialjob_cfx
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L ansys@osc:1

#Set up CFX environment.
module load ansys
#Copy CFX files like .def to $TMPDIR and move there to execute the program
cp test.def $TMPDIR/
cd $TMPDIR
#Run CFX in serial with test.def as input file
cfx5solve -batch -def test.def 
#Finally, copy files back to your home directory
cp  * $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the job.txt  file with the command: sbatch job.txt  

Non-interactive Batch Job (Parallel Execution using HPC token)

CFX can be run in parallel, but it is very important that you read the documentation in the CFX Manual on the details of how this works.

In addition to requesting the base license token ( -L ansys@osc:1 ), you need to request copies of the ansyspar license, i.e., HPC tokens ( -L ansys@osc:1,ansyspar@osc:[n] ), where [n] is equal to the number of cores you requested minus 4.

Parallel jobs have to be submitted on Cardinal via the batch system. An example of the batch script follows:

#!/bin/bash
#SBATCH --job-name=paralleljob_cfx
#SBATCH --time=10:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L ansys@osc:1,ansyspar@osc:52

#Set up CFX environment.
module load ansys
#Copy CFX files like .def to $TMPDIR and move there to execute the program
cp test.def $TMPDIR/
cd $TMPDIR
#Convert the node information into format for CFX host list
nodes=$(srun hostname | sort | \
uniq -c | \
awk '{print $2 "*" $1}' | \
paste -sd, -)
#Run CFX in parallel with new.def as input file
#if multiple nodes
cfx5solve -batch -def test.def  -par-dist $nodes -start-method "Platform MPI Distributed Parallel"
#if one node
#cfx5solve -batch -def test.def -par-dist $nodes -start-method "Platform MPI Local Parallel"
#Finally, copy files back to your home directory
cp  * $SLURM_SUBMIT_DIR

Further Reading

Supercomputer: 
Service: 

FLUENT

ANSYS FLUENT (called FLUENT hereafter) is a state-of-the-art computer program for modeling fluid flow and heat transfer in complex geometries.

Availability and Restrictions

FLUENT is available on the Cardinal Cluster. You can see the currently available versions in the table on the main Ansys page here.

You can use module spider ansys for Cardinal to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

Currently, there are in total 50 ANSYS base license tokens and 900 HPC tokens for academic users. These base tokens and HPC tokens are shared with all ANSYS products we have at OSC.  A base license token will allow FLUENT to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial FLUENT job with 1 core will need 1 base license token while a parallel FLUENT job with 28 cores will need 1 base license token and 24 HPC tokens.

Access for Commercial Users

Contact OSC Help for getting access to FLUENT if you are a commercial user.

Usage

Usage on Cardinal

Set-up on Cardinal

To load the default version of FLUENT module, use  module load ansys. To select a particular software version, use module load ansys/version. For example, use  module load ansys/17.2  to load FLUENT version 17.2 on Cardinal. 

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your FLUENT analysis to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

Interactive mode is similar to running FLUENT on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run FLUENT interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in non-interactive batch mode.

To run interactive FLUENT GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please following the steps below to use FLUENT GUI interactively:

  1. Ensure that your SSH client software has X11 forwarding enabled
  2. Connect to Cardinal system
  3. Request an interactive job. The command below will request one whole node with 28 cores ( -N 1 -n 28), for a walltime of one hour (-t 1:00:00), with one FLUENT license (modify as per your own needs):
    sinteractive -N 1 -n 28 -t 1:00:00 -L ansys@osc:1,ansyspar@osc:24
  4. Once the interactive job has started, run the following commands to setup and start the FLUENT GUI:

    module load ansys
    fluent 
    
Non-interactive Batch Job (Serial Run Using 1 Base Token)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

Below is the example batch script ( job.txt) for a serial run with an input file (run.input) on Cardinal:

#!/bin/bash
#SBATCH --job-name=serial_fluent
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L ansys@osc:1
#
# The following lines set up the FLUENT environment
#
module load ansys
#
# Copy files to $TMPDIR and move there to execute the program
#
cp test_input_file.cas test_input_file.dat run.input $TMPDIR
cd $TMPDIR
#
# Run fluent
fluent 3d -g < run.input  
#
# Where the file 'run.input' contains the commands you would normally
# type in at the Fluent command prompt.
# Finally, copy files back to your home directory
cp *   $SLURM_SUBMIT_DIR 

As an example, your run.input file might contain:

file/read-case-data test_input_file.cas 
solve/iterate 100
file/write-case-data test_result.cas
file/confirm-overwrite yes    
exit  
yes  

In order to run it via the batch system, submit the job.txt file with the command: sbatch job.txt 

Non-interactive Batch Job (Parallel Execution using HPC token)

FLUENT can be run in parallel, but it is very important that you read the documentation in the FLUENT Manual on the details of how this works.

In addition to requesting the FLUENT base license token (-L ansys@osc:1), you need to request copies of the ansyspar license, i.e., HPC tokens (-L ansys@osc:1,ansyspar@osc:[n]), where [n] is equal to the number of cores you requested minus 4.

Parallel jobs have to be submitted to Cardinal via the batch system. An example of the batch script follows:

#!/bin/bash
#SBATCH --job-name=parallel_fluent
#SBATCH --time=3:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L ansys@osc:1,ansyspar@osc:52
set echo on   
hostname   
#   
# The following lines set up the FLUENT environment   
#   
module load ansys
#      
# Create the config file for socket communication library   
#   
# Create list of nodes to launch job on   
rm -f pnodes   
cat  $PBS_NODEFILE | sort > pnodes   
export ncpus=`cat pnodes | wc -l`   
#   
#   Run fluent   
fluent 3d -t$ncpus -pinfiniband.ofed -cnf=pnodes -g < run.input 

Known Issues

Parallel job hang and startup failed

Resolution: Resolved with workaround
Update: April 2024
Version: All

FLUENT parallel jobs with default MPI (Intel MPI) may experience startup failures, leading to job hang due to a recent Slurm upgrade. Intel MPI in FLUENT uses SSH as the default bootstrap mechanism to launch the Hydra process manager. Starting with Slurm version 23.11, the environment variable I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher is added because Slurm is set as the default bootstrap system (I_MPI_HYDRA_BOOTSTRAP=slurm). However, this causes an issue when SSH is utilized as the bootstrap system.

Workaround

Prepend export -n I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS to the fluent command-line.

Reference

Further Reading

See Also

Supercomputer: 
Service: 

Workbench Platform

ANSYS Workbench platform is the backbone for delivering a comprehensive and integrated simulation system to users. See ANSYS Workbench platform for more information. 

Availability and Restrictions

ANSYS Workbench is available on Cardinal Cluster. You can see the currently available versions in the table on the main Ansys page here.

You can use module spider ansys  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

Access for Commercial Users

Contact OSC Help for getting access to ANSYS if you are a commercial user.

Usage

Usage on Cardinal

Set-up for Structural-Fluid dynamics related applications

To load the default version , use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/17.2   to load version 17.2 on Cardinal. After the module is loaded, use the following command to open Workbench GUI:

runwb2

Set-up for CFD related applications

To load the default version , use  module load ansys  . To select a particular software version, use   module load ansys/version   . For example, use  module load ansys/17.2   to load version 17.2 on Cardinal. After the module is loaded, use the following command to open Workbench GUI:

runwb2

Further Reading

See Also

Supercomputer: 
Service: 

AlphaFold 3

AlphaFold 3 developed by DeepMind and Isomorphic Labs, is an advanced artificial intelligence system that predicts the 3D structures of proteins and their interactions with other molecules, including DNA, RNA, ligands, and ions.

Availability and Restrictions

Versions

Version Ascend Cardinal
3.0.1 X X

You can use module spider alphafold3 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

AlphaFold 3 is available for all OSC users.

Publisher/Vendor/Repository and License Type

Copyright 2024 DeepMind Technologies Limited.

The AlphaFold 3 source code is licensed under the Creative Commons Attribution-Non-Commercial ShareAlike International License, Version 4.0 (CC-BY-NC-SA 4.0) (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://github.com/google-deepmind/alphafold3/blob/main/LICENSE.

Obtaining Model Parameters

The AlphaFold 3 model parameters are made available under the AlphaFold 3 Model Parameters Terms of Use (the "Terms"); you may not use these except in compliance with the Terms. You may obtain a copy of the Terms at https://github.com/google-deepmind/alphafold3/blob/main/WEIGHTS_TERMS_OF_USE.md.

Due to the Terms of Use, OSC will no longer maintain the model parameters in a central location. Users must download them by following the instructions at: https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#obtaining-model-parameters.

Usage

Batch Usage

Set-up

To load the default version of AlphaFold 3 module, use module load alphafold3.

Batch Usage

Below is the example batch script for an AlphaFold 3 job:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=32
#SBATCH --gpus-per-node=1

module reset
module load alphafold3/3.0.1

run_alphafold.sh --model_dir=/path/to/your/model/parameters \
                 --output_dir=$(pwd -P)/output \
                 --json_path=2PV7.json

To get full-options list

run_alphafold.sh --helpshort

Note that the recommended hardware for AlphaFold 3 includes H100 (Cardinal) and A100 (Ascend). There are known issues with V100 (Pitzer), and additional parameters are required to run AlphaFold 3 on Pitzer, as referenced in the following links:

Using the example above, you need to modify the job script with the additional parameters, as shown below:

export APPTAINERENV_XLA_FLAGS="--xla_disable_hlo_passes=custom-kernel-fusion-rewriter"  
run_alphafold.sh --model_dir=/path/to/your/model/parameters \
                 --output_dir=$(pwd -P)/output \
                 --json_path=2PV7.json \
                 --flash_attention_implementation=xla

Best Practice

Request correct number of CPUs for multiple sequence alignments (MSA)

An AlphaFold 3 run with a single protein sequence launches four parallel JackHMMER processes, each requesting eight worker threads. It is recommended to request 32 CPUs per node per job, e.g., --ntasks-per-node=32.

Further Reading

Tag: 
Supercomputer: 
Service: 
Fields of Science: 

AlphaFold

AlphaFold is a software package that provides an implementation of the inference pipeline of AlphaFold v2.0. This is a completely new model that was entered in CASP14 and pusblished in Nature.

Availability and Restrictions

Versions

Version Ascend Cardinal Model Parameters
2.0.0     2021-07-14
2.1.0   X 2021-10-27
2.1.2   X 2022-01-19
2.2.2     2022-03-02; Multimer model weights: v2
2.2.4   X 2022-03-02; Multimer model weights: v2
2.3.1     2022-12-06; Multimer model weights: v3
2.3.2 X X 2022-12-06; Multimer model weights: v3
* Current default version

You can use module spider alphafold to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

AlphaFold is available for all OSC users

Publisher/Vendor/Repository and License Type

Copyright 2021 DeepMind Technologies Limited

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See hte License for the specific language governing permissions and limitations under the License. See the License for specific langauge governing permissions and limitations under the License.

The AlphaFold parameters are made available for non-commercial use only, under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. You can find details at: https://creativecommons.org/licenses/by-nc/4.0/legalcode.

Usage

Usage on Pitzer

Set-up

To load the default version of AlphaFold module, use module load alphafold.

Batch Usage

Below is the example batch script (job.txt) for an alphafold job:

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --gpus-per-node=1
#SBATCH --gpu_cmode=shared


module reset
module load alphafold/2.1.2

run_alphafold.sh --use_gpu_relax=True --db_preset=reduced_dbs --fasta_paths=rcs_pdb_6VR4.fasta --max_template_date=2020-05-14 --output_dir=$(pwd -P)/output

The control options and presets for model and database:

Option Preset Note
--model_preset monomer
monomer_casp14
monomer_ptm
multimer
Control which AlphaFold model to run
--db_preset full_dbs
reduced_dbs

Control MSA speed/quality tradeoff

To get full-options list

run_alphafold.sh --helpshort

For very large simulations use multiple GPUs and to make sure a job can access all the GPU memory, set this before run_alphafold.sh with alphafold/2.2.2:

export TF_FORCE_UNIFIED_MEMORY=1
run_alphafold.sh ...

Note also that not all models are parallelized over multiple GPUs; see https://github.com/deepmind/alphafold/issues/30

Use custom AlphaFold

From 2.1.2 to 2.2.2, you can use own copy of AlphaFold code with our pre-installed packages and database. For example, you download a copy of AlphaFold 2.2.2 in $HOME/alphafold and make some changes. Modify the ALPHAFOLD_HOME variable before calling run_alphafold.sh, e.g.

module reset
module load alphafold/2.2.2

export ALPHAFOLD_HOME=$HOME/alphafold
run_alphafold.sh --db_preset=reduced_dbs --fasta_paths=rcs_pdb_6VR4.fasta --max_template_date=2020-05-14 --output_dir=$(pwd -P)/output

Batch Usage (2.0.0)

Below is the example batch script (job.txt) for an alphafold job:

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --gpus-per-node=2

module reset
module load alphafold/2.0.0

run_alphafold.sh --preset=reduced_dbs --fasta_paths=rcs_pdb_6VR4.fasta --max_template_date=2020-05-14 --output_dir=$(pwd -P)/output

Other available job options are:

--preset=recued_dbs, --preset=full_dbs, or --preset=casp14

 

Further Reading

Online documentation is available on the AlphaFold homepage.

Notes on AlphaFold output.

Notes on citing AlphaFold.

Tag: 
Supercomputer: 
Service: 
Fields of Science: 

Altair HyperWorks

HyperWorks is a high-performance, comprehensive toolbox of CAE software for engineering design and simulation.

Availability & Restrictions

Versions

The following version of Altair Hyperworks can be found for the following environments:

Version
13
2017.1
2019.2
2020.0
* Current Default Version

You can use module spider hyperworks to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HyperWorks is available to all academic clients. Please contact OSC Help to request the appropriate form for access.

Publisher/Vendor/Repository and License Type

Altair Engineering, Commercial (state-wide)

Usage

Using HyperWorks through OSC installation

To use HyperWorks on the OSC clusters, first ensure that X11 forwarding is enabled as the HyperWorks workbench is a graphical application. Then, load the hyperworks module:

module load hyperworks

The HyperWorks GUI can be launched then with the following command:

hw

The Hypermesh GUI can be launched then with the following command:

hm

State-wide access for HyperWorks

For information on downloading and installing a local copy through the state-wide license, follow the steps below. The versions of HyperWorks available statewide differ from the versions available at OSC on the OSC clusters. To check for the available statewide versions, complete steps 1 through 5 below.

NOTE: To run Altair HyperWorks, your computer must have access to the internet. The software contacts the license server at OSC to check out a license when it starts and periodically during execution. The amount of data transferred is small, so network connections over modems are acceptable.

 

Usage of HyperWorks on a local machine using the statewide license will vary from installation to installation.

  1. Go to https://altairone.com/home

  2. If you have already registered with the Altair website, click on "Sign In" in the upper right hand corner of the page, enter the e-mail address that you registered with and your password and skip to step #4. Otherwise click the "Sign Up" button instead and continue with step #3.

  3. You will be prompted for some contact information and an e-mail address which will be your unique identifier.

    • IMPORTANT: The e-mail address you give must be from your academic institution. Under the statewide license agreement, registration from Ohio universities is allowed on the Altair web site. Trying to log in with a yahoo or hotmail e-mail account will not work. If you enter your university e-mail and the system will not register you, please contact OSChelp at oschelp@osc.edu.

  4. Once you have logged in, go back to the home page and click on the button labeled "Altair Marketplace", where you can then press the button "Browse the Marketplace" which takes you to the Marketplace page.

  5. From here, you can search for the app you would like to use, in this case you're looking for the one listed as "HyperWorks" which you can search for in the search bar at the upper left corner of the Marketplace page.

  6. To download, you just need to press the "Download" button that appears in the side window that pops up after selecting the HyperWorks application from the marketplace page. From there you need to select the version you'd like and the target operating system for which it will run on. Then press the button that looks like an arrow point down at a "U" (aka Download symbol). In addition to downloading the software, download the "Installation Guide and Release Notes" for instructions on how to install the software.

    • NOTE: If you are a student and you click on the HyperWorks application in the marketplace, after creating an account and logging in, but see a "Try Now" button instead of a "Download" button then you may have not been added to the university account correctly (A known issue). To remedy this, please email support@altair.com with your name plus email, and ask the support team to update the account permissions so you can download the software.

    • IMPORTANT: If you have any questions or problems, please contact OSChelp at oschelp@osc.edu, rather than HyperWorks support. The software agreements outlines that problems should first be sent to OSC. If the OSC support line cannot answer or resolve the question, they have the ability to raise the problem to Altair support. If you have any general questions, or are looking for answers to frequently asked questions, you can check the Community Forums page for possible answers or help. But if you have problems, make sure to extend them to OSC first as stated above.

  7. Please contact OSC Help for further instruction and license server information. In order to be added to the allowed list for the state-wide software access, we will need your IP address/range of machine that will be running this software. 

  8. You need to set an environment variable (ALTAIR_LICENSE_PATH) on your local machine to point at our license server (7790@license6.osc.edu). See this link for instructions if necessary.

Further Reading

For more information about HyperWorks, see the following:

See Also

Service: 
Fields of Science: 

Apptainer (formerly Singularity)

Apptainer (formerly Singularity) is a container system designed for use on High Performance Computing (HPC) systems. It allows users to run both Docker and Singularity containers.

From the Docker website: "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

On June 21th, 2022, Singularity is replaced with Apptainer, which is just a renamed open-source project to avoid conflicts with SingularityCE so it can be accepted into the Linux Foundation. Apptainer 1.0 has the same code as Singularity after versions 3.8.x, and still provides the command singularity (apptainer is the official command). Thus, user should continue running containers on OSC systems without any issue: 

1. Containers built with Apptainer will continue to work with installations of Singularity.
2. User will see warnings about SINGULARITY_ and SINGULARITYENV_ environment variables.
    A future version of Apptainer may stop supporting environment variable compatibility so we recommned
    users to add respective APPTAINER_ and APPTAINERENV_ counterparts in their job environments.

For more detail, pleae visit the Singularity Compatibility page.

If you experience issues using Singularity after downtime, please contact OSC help.

Availability and Restrictions

Versions

Apptainer/Singularity is available on all OSC clusters. Only one version is available at any given time. To find out the current version:

apptainer version

Check the release page for the changelog: https://github.com/apptainer/apptainer/releases

Access

Apptainer/Singularity is available to all OSC users.

Publisher/Vendor/Repository and License Type

Apptainer project, established as Apptainer a Series of LF Projects LLC; 3-clause BSD License

Usage

Set-up

No setup is required. You can use Apptainer/Singularity directly on all clusters.

Using Apptainer/Singularity

See HOWTO: Use Docker and Singularity Containers at OSC for information about using Apptainer/Singularity on all OSC clusters, including some site-specific caveats.  

Example:  Run a container from the Singularity hub

[pitzer-login01]$ apptainer run shub://singularityhub/hello-world
INFO:    Downloading library image
Tacotacotaco
If unsure about the amount of memory that a apptainer process will require, then be sure to request an entire node for the job. It is common for singularity jobs to be killed by the OOM killer because of using too much RAM.


Known Issues

Workshop

Further Reading

Supercomputer: 
Service: 

AutoDock

AutoDock is a a suite of automated docking tools. It is designed to predict how small molecules, such as substrates or drug candidates, bind to a receptor of known 3D structure. AutoDock has applications in X-ray crystallography, structure-based drug design, lead optimization, etc.

Availability and Restrictions

Versions

AutoDock and AutoDock-GPU are available on the Cardinal Cluster. The versions currently available at OSC are:

AutoDock Cardinal
4.2.6 X

 

AutoDock-GPU Cardinal
1.5.2 X

 

You can use module spider autodock and module spider autodock-gpu to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of AutoDock is open to all OSC users. Please contact OSC Help for any questions.

Publisher/Vendor/Repository and License Type

Center for Computational Structural Biology, Open source

Usage

Usage on Cardinal

Set-up

To load the default version of AutoDock module, use module load autodock/4.2.6. Alternatively to load the accelerated version of AutoDock4 with GPU, use module load autodock-gpu/1.5.2.

Running AutoDock

AutoDock executables can be run as documented in the AutoDock User Manual.

Running AutoDock jobs with GPU

A GPU can be utilized for AutoDock. You can acquire a GPU for the job by

#SBATCH --gpus-per-node=1

If running with an OnDemand desktop, select a GPU node to launch the desktop on.  For more detail, please read here.

 

For more information about GPU computing with AutoDock, please read AutoDock-GPU wiki.

Further Reading

 

Supercomputer: 
Service: 

BCFtools

BCFtools is a set of utilities that manipulate variant calls in the Variant Call Format (VCF) and its binary counterpart BCF.

Availability and Restrictions

Versions

The following versions of BCFtools are available on OSC clusters:

Version Ascend Cardinal
1.17 X X
1.21 X X

 

You can use module spider bcftools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

BCFtools is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Genome Research Ltd., Open source

Usage

Set-up

To configure your environment for use of BCFtools, run the following command: module load bcftools/version. For example, use  module load bcftools/1.17to load BCFtools 1.17.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

BLAS

The BLAS (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations.

Availability and Restrictions

Access

A highly optimized implementation of the BLAS is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building the BLAS for yourself. MKL is available to all OSC users.

Usage

See OSC's MKL software page for usage information. Note that there is no library named libblas.a or libblas.so. The flag "-lblas" on your link line will not work. You should modify your makefile or build script to link to the MKL libraries instead.

Further Reading

Service: 
Technologies: 
Fields of Science: 

BLAST

The BLAST programs are widely used tools for searching DNA and protein databases for sequence similarity to identify homologs to a query sequence. While often referred to as just "BLAST", this can really be thought of as a set of programs: blastp, blastn, blastx, tblastn, and tblastx.

Availability & Restrictions

Versions

The following versions of BLAST are available on OSC systems: 

Version Pitzer Ascend Cardinal
2.16.0 X X X

 

You can use module spider blast-plus to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

BLAST is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

National Institutes of Health, Open source

Usage

Set-up

To load BLAST, type the following into the command line:

module load blast-plus/2.16.0

Then create a resource file .ncbirc, and put it under your home directory.

Using BLAST

The five flavors of BLAST mentioned above perform the following tasks:

  • blastp: compares an amino acid query sequence against a protein sequence database

  • blastn: compares a nucleotide query sequence against a nucleotide sequence database

  • blastx: compares the six-frame conceptual translation products of a nucleotide query sequence (both strands) against a protein sequence database

  • tblastn: compares a protein query sequence against a nucleotide sequence database dynamically translated in all six reading frames (both strands).

  • tblastx: compares the six-frame translations of a nucleotide query sequence against the six-frame translations of a nucleotide sequence database. (Due to the nature of tblastx, gapped alignments are not available with this option)

NCBI BLAST Database

Information on the NCBI BLAST database can be found here. https://www.osc.edu/resources/available_software/scientific_database_list/blast_database 

We provide local access to nt and refseq_protein databases. You can access the database by loading desired blast-database modules. If you need other databases, please send a request email to OSC Help .

Batch Usage

A sample batch script on Pitzer is below:

#!/bin/bash
## --ntasks-per-node can be increased upto 48 on Pitzer
#SBATCH --nodes=1 --ntasks-per-node=28 
#SBATCH --time=00:10:00
#SBATCH --job-name Blast
#SBATCH --account=<project-account>

module load blast
module load blast-database/2018-08

cp 100.fasta $TMPDIR
cd $TMPDIR

tblastn -db nt -query 100.fasta -num_threads 16 -out 100_tblastn.out

cp 100_tblastn.out $SLURM_SUBMIT_DIR

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

BWA

BWA is a software package for mapping low-divergent sequences against a large reference genome, such as the human genome. It consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM.

Availability and Restrictions

Versions

The following versions of BWA are available on OSC clusters:

Version Pitzer Cardinal
0.7.17 X  
0.7.18   X
* Current default version

You can use module spider bwa to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

BWA is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Li H. and Durbin R., Open source

Usage

Set-up

To configure your environment for use of BWA, run the following command: module load bwa/version. For example, use  module load bcftools/0.7.17to load BWA 0.7.17.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Blender

Blender is the free and open source 3D creation suite. It supports the entirety of the 3D pipeline—modeling, rigging, animation, simulation, rendering, compositing and motion tracking, even video editing and game creation.

Availability and Restrictions

Versions

The following versions of Blender are available on OSC systems: 

Version Pitzer Ascend Cardinal
4.2 X X X

You can use module spider blender to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Blender is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Blender Foundation, Open source

Usage

Set-up for Blender 4.2

module load blender/4.2

Using Blender 4.2

Preferred: Select Blender from OnDemand interactive apps and choose version 4.2 from the drop-down menu.

Alternative: Open a Pitzer desktop, selecting 'vis' node.  Load the module and run

blender

from the command line.

Further Reading

 

Tag: 
Supercomputer: 
Service: 
Fields of Science: 

Boost

Boost is a set of C++ libraries that provide helpful data structures and numerous support functions in a wide range of aspects of programming, such as, image processing, gpu programming, concurrent programming, along with many algorithms.  Boost is portable and performs well on a wide variety of platforms.

Availability & Restrictions

Versions

The following version of Boost are available on OSC systems:

Version Pitzer Ascend Cardinal Notes
1.83.0 X(GI) X(GI) X(GI)  
G = available with gnu; I = available with intel

You can use module spider boost to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Boost is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Beman Dawes, David Abrahams, Rene Rivera/ Open source

Usage

Usage

Set-up

Initalizing the system for use of the Boost library is independent of the compiler you are using.  To load the boost module run the following command:

module load boost/1.83.0

Building With Boost

The following environment variables are setup when the Boost library is loaded:

VARIABLE USE
$BOOST_CFLAGS Use during your compilation step for C++ programs.
$BOOST_LIBS Use during your link step.

 

Below is a set of example commands used to build and run a file called  example2.cpp. First copy the example2.cpp and jayne.txt from Oakley into your home directory with the following commands:

cp /usr/local/src/boost/boost-1_56_0/test.osc/example2.cpp ~
cp /usr/local/src/boost/boost-1_56_0/test.osc/jayne.txt ~
Then compile and test the program with the folllowing commands:
g++ $BOOST_CFLAGS example2.cpp -o boostTest $BOOST_LIBS -lboost_regex
./boostTest < jayne.txt

Further Reading

 

Supercomputer: 
Service: 
Fields of Science: 

Bowtie

Bowtie is an ultrafast, memory-efficient short read aligner. It aligns short DNA sequences (reads) to the human genome at a rate of over 25 million 35-bp reads per hour. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small: typically about 2.2 GB for the human genome (2.9 GB for paired-end).

Availability and Restrictions

Versions

The following versions of Bowtie1 are available on OSC clusters:

Version Pitzer Ascend Cardinal
1.3.1 X X X*
* Current default version

You can use module spider bowtie to view available modules for a given cluster. Feel free to contact OSC Help if you need other versions for your work.

Access

Bowtie1 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Ben Langmead et al., Open source (Artistic 2.0)

Usage

Usage on Pitzer

Set-up

To configure your environment for use of Bowtie1, run the following command:  module load bowtie/1.3.1.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Bowtie2

Bowtie2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. It is particularly good at aligning reads of about 50 up to 100s or 1,000s of characters, and particularly good at aligning to relatively long (e.g. mammalian) genomes. Bowtie 2 indexes the genome with an FM Index to keep its memory footprint small: for the human genome, its memory footprint is typically around 3.2 GB. Bowtie 2 supports gapped, local, and paired-end alignment modes.

Please note that bowtie (and tophat) CANNOT run in parallel, that is, on multiple nodes.  Submitting multi-node jobs will only waste resources.  In addition you must explicitly include the '-p' option to use multiple threads on a single node.

Availability and Restrictions

Versions

The following versions of Bowtie2 are available on OSC clusters:

Version Pitzer Ascend Cardinal Note
2.5.1 X X X*  
* Current default version

You can use module spider bowtie2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Bowtie2 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Ben Langmead et al., Open source

Usage

Usage

Set-up

To configure your environment for use of Bowtie2, run the following command: module load bowtie2/2.5.1.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

CMake

CMake is a family of compilation tools that can be used to build, test and package software.

Availability and Restrictions

Versions

The current versions of CMake available at OSC are:

Version Pitzer Ascend Cardinal
3.25.2 X X X*
3.26.5 X# X# X#
* Current default version; # System version

You can use  module spider cmake  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

CMake is available to all OSC users.

Publisher/Vendor/Repository and License Type

Aaron C. Meadows et al., Open source

Usage

Usage

Set-up

To configure your environment for use of BCFtools, run the following command: module load cmake/version. For example, use  module load bcftools/2.25.2to load cmake 2.25.2.

Further Reading

For more information, visit the CMake homepage.

Supercomputer: 

COMSOL

COMSOL Multiphysics (formerly FEMLAB) is a finite element analysis and solver software package for various physics and engineering applications, especially coupled phenomena, or multiphysics. owned and supported by COMSOL, Inc.

Availability and Restrictions

Versions

The versions currently available at OSC are:

Version Cardinal
6.2 X
* Current default version

You can use module spider comsol  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

COMSOL is for academic use, available only to the Ohio State University users. OSC does not provide COMSOL licenses for academic use to students and faculty outside of the Ohio State University due to licensing restrictions. If you or your institution have a network COMSOL license server, you may be able to use it on OSC. For connections to your license server from OSC, please read this document. If you need further help, please contact OSC Help.

To use COMSOL you will have to be added to the license server.  Please contact OSC Help to be added.

Access for Commercial Users

Contact OSC Help for getting access to COMSOL if you are a commercial user. 

Publisher/Vendor/Repository and License Type

Comsol Inc., Commercial

Usage

Usage on Cardinal

Set-up

To load the default version of COMSOL module, use  module load comsol . To select a particular software version, use   module load comsol/version . For example, use  module load comsol/52a  to load COMSOL version 5.2a. 

Batch Usage

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session, one can run the following command:
sinteractive -A <project-account> -N 1 -n 28 -t 1:00:00 -L comsolscript@osc:1
which gives you 28 cores ( -N 1 -n 28 ) with 1 hour ( -t 1:00:00 ). You may adjust the numbers per your need.
Non-interactive Batch Job (Serial Run)

Assume that you have had a comsol script file  mycomsol.m  in your working direcory ( $SLURM_SUBMIT_DIR ). Below is the example batch script ( job.txt ) for a serial run: 

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L comsolscript@osc:1
#SBATCH --account=<project-account>
#
# The following lines set up the COMSOL environment
#
module load comsol
#
# Use TMPDIR for best performance
cp -p mycomsol.m $TMPDIR
cd $TMPDIR
#
# Run COMSOL
#
comsol batch mycomsol
#
# Now, copy data (or move) back once the simulation has completed
#
cp -p * $SLURM_SUBMIT_DIR
Non-interactive Batch Job (Parallel Run for COMSOL 6.0 and Later)

Below is the example batch script for a parallel job using COMSOL 6.0 or later versions:

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=2 --ntasks-per-node=4 --cpus-per-task=7
#SBATCH -L comsolscript@osc:1
#SBATCH --account=<project-account>

module load comsol
echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
cp input_cluster.mph $TMPDIR
cd $TMPDIR

echo "--- COMSOL run"
comsol batch -mpibootstrap slurm -inputfile input_cluster.mph -outputfile output_cluster.mph
echo "--- Copy files back"
cp output_cluster.mph output_cluster.mph.status ${SLURM_SUBMIT_DIR}
echo "---Job finished at: 'date'"
echo "---------------------------------------------"

Note:

  • Use the "-mpibootstrap slurm" option to take the resource specification from the SBATCH directives, thus eliminating the -nnhost, -nn, and -np options.  For more details see https://www.comsol.com/support/knowledgebase/1001
  • Copy files from your directory to $TMPDIR.
  • Provide the name of the input file and output file.
OLD Non-interactive Batch Job (Parallel Run for COMSOL 4.3 and Later)

As of version 4.3, it is not necessary to start up MPD before launching a COMSOL job. Below is the example batch script ( job.txt ) for a parallel run using COMSOL 4.3 or later versions:

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L comsolscript@osc:1
#SBATCH --account=<project-account>

module load comsol
echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
cp input_cluster.mph $TMPDIR
cd $TMPDIR

echo "--- COMSOL run"
comsol -nn 2 batch -mpirsh ssh -inputfile input_cluster.mph -outputfile output_cluster.mph
echo "--- Copy files back"
cp output_cluster.mph output_cluster.mph.status ${SLURM_SUBMIT_DIR}
echo "---Job finished at: 'date'"
echo "---------------------------------------------"

Note:

  • Set nodes to 2 and ppn to 28 ( --nodes=2 --ntasks-per-node=28). You can change the values per your need.
  • Use "-mpirsh ssh" option for multi-node jobs
  • Copy files from your directory to $TMPDIR.
  • Provide the name of the input file and output file.

Avaliable COMSOL modules with OSC's academic license

Note: Last updated 02/05/24

AC/DC Module
Battery Design Module
CAD Import Module
CFD Module
Chemical Reaction Engineering Module
Heat Transfer Module
LiveLink for MATLAB
MEMS Module
Microfluidics Module
Particle Tracing Module
RF Module
Semiconductor Module
Structural Mechanics Module
Subsurface Flow Module

Further Reading

Supercomputer: 
Service: 

Interactive Parallel COMSOL Job

This documentation is to discuss how to set up an interactive parallel COMSOL job at OSC. The following example demonstrates the process of using COMSOL version 5.1 on Oakley. Depending on the version of COMSOL and cluster you work on, there mighe be some differences from the example. Feel free to contact OSC Help if you have any questions. 

  • Launch COMSOL GUI application following the instructions on this page. Get the information on the node(s) allocated to your job and save it in the file named hostfile using the following command:

 

cat $PBS_NODEFILE | uniq > hostfile

Make sure the hostfile is located in the same directory where you COMSOL input file is put

  • Open COMSOL GUI application. To enable the cluster compuitng feature, click the show button and select Advanced Study Options, as shown in the picture below:

Advanced study

  • In the Model Builder, right-click a Study node and select Cluster Computing, as shown in the picture below:

cluster computing

  • In the Cluster Computing node's setting window, select General from the Cluster type list. Provide the name of Host file as hostfile. Browse to the directory where your COMSOL input file is located, as shown in the picture below:

setting

  • Save all the settings. Then you should be able to run an interactive parallel COMSOL job at OSC
Supercomputer: 

CP2K

CP2K is a quantum chemistry and solid state physics software package that can perform atomistic simulations of solid state, liquid, molecular, periodic, material, crystal, and biological systems. CP2K provides a general framework for different modeling methods such as DFT using the mixed Gaussian and plane waves approaches GPW and GAPW. Supported theory levels include DFTB, LDA, GGA, MP2, RPA, semi-empirical methods and classical force fields. CP2K can do simulations of molecular dynamics, metadynamics, Monte Carlo, Ehrenfest dynamics, vibrational analysis, core level spectroscopy, energy minimization, and transition state optimization using NEB or dimer method.

Availability and Restrictions

Versions

CP2K is available on the OSC clusters. These are the versions currently available:

VERSION Pitzer Ascend Cardinal Notes
2023.2 X X X  gcc/12.3.0 openmpi/5.0.2
2023.2-openblas X X X  gcc/12.3.0 openmpi/5.0.2

You can use module spider cp2k to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

CP2K is available to all OSC users.

Publisher/Vendor/Repository and License Type

CP2K, GNU General Public License

Usage

IMPORTANT NOTE: You need to load the prerequisite compiler and MPI modules before you can load CP2K. To determine those modules, use module spider cp2k/{version}.

Usage

Set-up

CP2K usage is controlled via modules. Load one of the CP2K modulefiles at the command line, in your shell initialization script, or in your batch scripts. You need to load the prerequisite compiler and MPI modules before you can load CP2K. To determine those modules, use, e.g.: module spider cp2k/2023.2

Batch Usage

When you log into pitzer.osc.edu you are actually logged into the login node. To gain access to the vast resources in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

For an interactive batch session one can run the following command:

sinteractive -A <project-account> -n 1 -t 00:20:00

which requests one core (-n 1), for a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

Non-interactive Batch Job

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script for a parallel run:

#!/bin/bash 
#SBATCH --nodes=2
#SBATCH --time=1:00:0
#SBATCH --account=<project-account>
#SBATCH --gres=pfsdir

module load  gcc/12.3.0 openmpi/5.0.2
module load  cp2k/2023.2-openblas
module list
module help  cp2k/2023.2-openblas 

cp job.inp $PFSDIR/job.inp
cd $PFSDIR
srun cp2k.popt -i job.inp -o job.out.$SLURM_JOB_ID 
cp job.out.$SLURM_JOB_ID $SLURM_SUBMIT_DIR/job.out.$SLURM_JOB_ID

This script uses the Scratch storage system, which is designed to synchronize storage across nodes temporarily, more information is available under the storage documentation in the "Further reading" section.

Known Issues

CP2K/2023.2 can produce huge output containing MKL messages

Further Reading

General documentation is available from the CP2K website.
Scratch Storage documentation is available from the Storage Guide

 
Supercomputer: 
Service: 

CUDA

CUDA™ (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by Nvidia that enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU).

Availability and Restrictions

Versions

CUDA is available on the clusters supporting GPUs. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal cuDNN library
11.8.0 X X X 8.8.1
12.1.1     X  
12.2.2     X  
12.3.2     X  
12.4.1 X X X*  
12.6.2 X X X  
12.8.1   X    
* Current default version
From CUDA 11 onwards, applications compiled with a CUDA major release can have minor version compatibility, meaning you may run a CUDA 11 application with any CUDA 11.x toolkit. See https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compati... for more detail.

You can use module spider cuda to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

CUDA is available for use by all OSC users.

Publisher/Vendor/Repository and License Type

Nvidia, Freeware 

Usage

Usage on Pitzer

Set-up on Pitzer

To load a CUDA version module, use module spider cuda then module load cuda/{version}.

GPU Computing SDK

The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. 

Programming in CUDA

Please visit the following link to learn programming in CUDA, http://developer.nvidia.com/cuda-education-training. The link also contains tutorials on optimizing CUDA codes to obtain greater speedups.

Compiling CUDA Code

Many of the tools loaded with the CUDA module can be used regardless of the compiler modules loaded. However, CUDA codes are compiled with nvcc, which depends on the GNU compilers. In particular, if you are trying to compile CUDA codes and encounter a compiler error such as

#error -- unsupported GNU version! gcc versions later than X are not supported!

then you need to load an older GNU compiler with the module load gnu/version command (if compiling standard C code with GNU compilers) or the module load gcc-compatibility/version command (if compiling standard C code with Intel or PGI compilers).

One can type module show cuda-version-number to view the list of environment variables.
To compile a cuda code contained in a file, let say mycudaApp.cu, the following could be done after loading the appropriate CUDA module: nvcc -o mycudaApp mycudaApp.cu. This will create an executable by name mycudaApp.

The environment variable OSC_CUDA_ARCH defined in the module can be used to specify the CUDA_ARCH, to compile with nvcc -o mycudaApp -arch=$OSC_CUDA_ARCH mycudaApp.cu.

Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

Debugging CUDA code

cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

Detecting memory access errors

CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

Setting the GPU compute mode on Pitzer

The GPUs on Pitzer can be set to different compute modes as listed here.

The default compute mode is the default setting on our GPU nodes (--gpu_cmode=shared), so you don't need to specify if you require this mode. With this mode, mulitple CUDA processes across GPU nodes are allowed, e.g CUDA processes via MPI. So, if you need to run a MPI-CUDA job, just keep the default compute mode. Should you need to use another compute mode, use --gpu_cmode to specify the mode setting. For example:

--nodes=1 --ntasks-per-node=40 --gpus-per-node=1 --gpu_cmode=exclusive

Batch Usage on Pitzer

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session

For an interactive batch session one can run the following command:

sinteractive -A <project-account> -N 1 -n 40 -g 2 -t 00:20:00

which requests one whole node (-N 1), 40 cores (-n 40), 2 gpus (-g 2), and a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (job.txt) for a serial run:

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --nodes=1 --ntasks-per-node=1 --gpus-per-node=1
#SBATCH --job-name Compute
#SBATCH --account=<project-account>

module load cuda
cd $HOME/cuda
cp mycudaApp $TMPDIR
cd $TMPDIR
./mycudaApp

CUDA Architecture

As mentioned in the previous Usage sections, to ensure that your the application you build runs regardless of changes to CUDA drivers, make sure you specify the architecture at runtime. You can use the helper OSC_CUDA_ARCH environment variable defined the cuda module to build your applications nvcc -o mycudaApp -arch=$OSC_CUDA_ARCH mycudaApp.cu.

Compiler and CUDA arch Support for NVCC 

Note that as of summer 2025 OSC GPUs have SM architectures of 70, 80, and 90 for V100, A100, and H100.

CUDA Version Supported SM arch Max supported GCC version Max supported Intel version Max supported oneAPI version
9.2.88 - 10.0.130 30-70 7    
10.1.168 - 10.2.89 30-75 8    
11.0 50-80 9    
11.1 - 11.4.0 50-80 10    
11.4.1 - 11.8 50-90 11    
12.0 60-90 12.1    
12.1 - 12.3 60-90 12.2 2021.10.0 2023.1.0
12.4 - 12.6 60-90 13.2    

Further Reading

Online documentation is available on the CUDA homepage.

Compiler support for the latest version of CUDA is available here.

CUDA optimization techniques.

 

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Cell Ranger

Cell Ranger is a cell analysis library for generate feature-barcode matrices, perform Analysis for RNA samples. Cell Ranger works in pipelines for it's RNA  sequencing analysis which allows it to: process raw sequencing output, read alignment, generate gene-cell matrices, and can perform downstream analyses such as clustering and gene expression analysis.

Availability and Restrictions

Versions

Cell Ranger is available on the Ascend Cluster. The versions currently available at OSC are:

Version Ascend Notes
7.2.0 X  

You can use module spider cellranger to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Cell Ranger is available to only academic OSC users. Please review the license agreement and 10x Privacy Policy before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

The 10x Genomics group, Closed source (academic)

Usage

Usage on Ascend

Set-up

To configure your environment for use of Cell Ranger, run the following command:  module load cellranger/version. For example, use module load cellranger/7.2.0 to load Cell Ranger 7.0.0.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Connectome Workbench

Connectome Workbench is an open source, freely available visualization and analysis tool for neuroimaging data, especially data generated by the Human Connectome Project.

Availability and Restrictions

Versions

Connectome Workbench is available on Pitzer and Cardinal clusters. These are the versions currently available:

Version Pitzer Ascend Cardinal Notes
1.3.2   X X  
1.5.0 X   X  
2.0.0   X X*  
* Current default version

You can use module spider connectome-workbench to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Connectome Workbench is available to all OSC users.

Publisher/Vendor/Repository and License Type

Washington University School of Medicine, GPL

Usage

Set-up

To configure your environment for use of the workbench, run the following command:  module load connectome-workbench/version virtualgl/2.6.5. The default version will be loaded; the virtualgl module is required as well on some platforms. For example, use  module load connectome-workbench/1.3.2 to load Connectome Workbench 1.3.2.

Further Reading

General documentation is available from the Connectome Workbench hompage.

 
Supercomputer: 
Service: 
Fields of Science: 

Cufflinks

Cufflinks is a program that analyzes RNA -Seq samples. It assembles aligned RNA-Seq reads into a set of transcripts, then inspects the transcripts to estimate abundances and test for differential expression and regulation in the RNA-Seq reads.

Availability and Restrictions

Versions

Cufflinks is available on the Cardinal Cluster. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2.2.1 X X X*
* Current Default Version

You can use module spider cufflinks to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Cufflinks is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Cole Trapnell et al., Open source

Usage

Further Reading

Supercomputer: 
Fields of Science: 

DS9

SAOImageDS9 is a astronomical imaging and data visualization application. DS9 provides support for FITS images, binary tables, multiple frame buffers, region manipulation, and colormaps display options

Availability and Restrictions

DS9 is currently available on the following clusters.

Version Pitzer Ascend Cardinal
8.6 X* X* X*
* Current default version

You can use  module spider ds9to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

DS9 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Jessica Mink, Smithsonian Astrophysical Observatory/ Open source

Usage

Usage

Set-up

To configure your environment for use of DS9, run the following command: module spider ds9. To load a particular DS9 version, use module load ds9/version . For example, use module load ds9/8.6to load DS9 8.6.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

DSI Studio

DSI Studio is a tractography software tool that maps brain connections and correlates findings with neuropsychological disorders. It is a collective implementation of several diffusion MRI methods, including diffusion tensor imaging (DTI), generalized q-sampling imaging (GQI), q-space diffeomorphic reconstruction (QSDR), diffusion MRI connectometry, and generalized deterministic fiber tracking.

Availability and Restrictions

The following versions of DSI Studio are available on OSC clusters:

Version Pitzer Ascend Cardinal
2024.June     X*
2025.Jan   X  
2025.Apr X    
* Current default version

You can use  module spider dsi-studio to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

DSI Studio is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

DSI Studio is free and licensing information for both academic and non-academic licenses is available at the DSI Studio homepage.

Please refer to the citation page about how to acknowledge DSI Studio.

Usage

Usage on Pitzer

Set-up

To configure your environment for use of DSI Studio, run the following command: module load dsi-studio. The default version will be loaded. To select a particular version, use module load dsi-studio/version. For example, use module load dsi-studio/2021.May to load DSI Studio 2.0. It is also recommended you use in conjunction with module load singularity.
 

DSI Studio is installed in a singularity container.  DSI_IMG environment variable contains the container image file path. So, an example usage would be

module load dsi-studio
singularity exec $DSI_IMG dsi_studio

This command will open the DSI Studio GUI environment, and we recommend Ondemand VDI or Desktop for GUI. 

For more information about singularity usages, please read OSC singularity page

 

Further Reading

 
Supercomputer: 
Service: 
Fields of Science: 

Darshan

Darshan is a lightweight "scalable HPC I/O characterization tool".  It is intended to profile I/O by emitting log files to a consistent log location for systems administrators, and also provides scripts to create summary PDFs to characterize I/O in MPI-based programs.

Availability and Restrictions

Versions

The following versions of Darshan are available on OSC clusters:

Version Pitzer Ascend Cardinal
3.4.5 X   X*
3.4.6   X  
* Current default version

You can use module spider darshan to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access 

Darshan is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

MCSD, Argonne National Laboratory, Open source

Usage

Usage on Pitzer

Setup

To configure the Pitzer cluster for Darshan run module spider darshan/VERSION to find supported compiler and MPI implementations, e.g.

$ module spider darshan/3.2.1

------------------------------------------------------------------------------------------------
  darshan: darshan/3.2.1
------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "darshan/3.2.1" module is available to load.

      intel/19.0.3  intelmpi/2019.7
      intel/19.0.3  mvapich2/2.3.1
      intel/19.0.3  mvapich2/2.3.2
      intel/19.0.3  mvapich2/2.3.3
      intel/19.0.3  mvapich2/2.3.4
      intel/19.0.3  mvapich2/2.3.5
      intel/19.0.5  intelmpi/2019.3
      intel/19.0.5  intelmpi/2019.7
      intel/19.0.5  mvapich2/2.3.1
      intel/19.0.5  mvapich2/2.3.2
      intel/19.0.5  mvapich2/2.3.3
      intel/19.0.5  mvapich2/2.3.4
      intel/19.0.5  mvapich2/2.3.5

then switch to the favorite programming environment and load the Darshan module:

$ module load intel/19.0.5 mvapich2/2.3.5
$ module load darshan/3.2.1

Batch Usage

Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations (Pitzer) and Scheduling Policies and Limits for more info. 

If you have an MPI-based program the syntax is as simple as

module load darshan

# basic call to darshan
export MV2_USE_SHARED_MEM=0
export LD_PRELOAD=$OSC_DARSHAN_DIR/lib/libdarshan.so
srun [args] ./my_mpi_program

# to show evidence that Darshan is working and to see internal timing
export DARSHAN_INTERNAL_TIMING=yes
srun [args] ./my_mpi_program
An Example of Using Darshan with MPI-IO

Below is an example batch script (darshan_mpi_pfsdir_test.sh) for testing MPI-IO and POSIX-IO.  Because the files generated here are large scratch files there is no need to retain them.

#!/bin/bash
#SBATCH --job-name="darshan_mpi_pfsdir_test"
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=2
#SBATCH --output=rfm_darshan_mpi_pfsdir_test.out
#SBATCH --time=0:10:0
#SBATCH -p parallel
#SBATCH --gres=pfsdir:ess

# Setup Darshan
module load intel
module load mvapich2
module load darshan
export DARSHAN_LOGFILE=${LMOD_SYSTEM_NAME}_${SLURM_JOB_ID/.*/}_${SLURM_JOB_NAME}.log
export DARSHAN_INTERNAL_TIMING=yes
export MV2_USE_SHARED_MEM=0
export LD_PRELOAD=$OSC_DARSHAN_DIR/lib/libdarshan.so

# Prepare the scratch files and run the cases
cp ~support/share/reframe/source/darshan/io-sample.c .
mpicc -o io-sample io-sample.c -lm
for x in 0 1 2 3; do  dd if=/dev/zero of=$PFSDIR/read_only.$x bs=2097152000 count=1; done
shopt -s expand_aliases
srun ./io-sample -p $PFSDIR -b 524288000 -v

# Generat report
darshan-job-summary.pl --summary $DARSHAN_LOGFILE

In order to run it via the batch system, submit the darshan_mpi_pfsdir_test.sh file with the following command:

sbatch darshan_mpi_pfsdir_test.sh

Further Reading

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Desmond

Desmond is a software package that perform high-speed molecular dynamics simulations of biological systems on conventional commodity clusters, general-purpose supercomputers, and GPUs. The code uses novel parallel algorithms and numerical techniques to achieve high performance and accuracy on platforms containing a large number of processors, but may also be executed on a single computer. Desmond includes code optimized for machines with an NVIDIA GPU.

Availability and Restrictions

Versions

The Desmond package is available on Ascend and Cardinal. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Note
2023.4   X X* GPU support only
2024.4 X X X GPU support only
* Current default version

You can use  module spider desmond to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.  Starting from the Desmond_Maestro_2019.1 release, desmond only supports GPUs.

Access for Academic Users 

Desmond is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.   Note that OSC has purchased and installed Schrödinger with paid licenses. This doesn't include the Desmond license. We have installed Desmond separately using free licenses.

Publisher/Vendor/Repository and License Type

D. E. Shaw Research, Non-Commercial

Usage

Usage on Ascend, Cardinal, and Pitzer

Set-up

To set up your environment for desmond load one of its module files:

​​module load desmond/2024.4

If you already have input and configuration files ready, here is an example batch script that uses Desmond non-interactively via the batch system:

#!/bin/bash
#SBATCH --job-name multisim-batch
#SBATCH --time=0:20:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH --account=<account>

# Example Desmond single-node batch script. 

sstat -j $SLURM_JOB_ID
module reset
module load desmond/2024.4
module list

sbcast -p desmondbutane.msj $TMPDIR/desmondbutane.msj
sbcast -p desmondbutane.cfg $TMPDIR/desmondbutane.cfg
sbcast -p desmondbutane.cms $TMPDIR/desmondbutane.cms

cd $TMPDIR
$SCHRODINGER/utilities/multisim -HOST localhost -maxjob 1 -cpu 24 -m desmondbutane.msj -c desmondbutane.cfg desmondbutane.cms -mode umbrella -ATTACHED -WAIT
ls -l
cd $SLURM_SUBMIT_DIR
sgather -r $TMPDIR $SLURM_SUBMIT_DIR

The WAIT option forces the multisim command to wait until all tasks of the command are completed. This is necessary for batch jobs to run effectively. The HOST option specifies how tasks are distributed over processors.

Set-up via Maestro

Desmond comes with its own Schrodinger interactive builder, Maestro. (Note that users should use matching versions of Desmond and Maestro, which is the case when following the details below; we have had reports of problems when mixing versions.) To run maestro, connect to OSC OnDemand and luanch a desktop, either via Desktops in the Interactive Apps drop down menu (these were labelled  Virtual Desktop Interface (VDI) previously) or via Shell Access in the Clusters drop down menu (these were labelled  Interactive HPC Desktop previously).  Click "Setup process" below for more detailed instructions.  Note that one cannot launch desmond jobs in maestro via the Schrodinger GUI in the Interactive Apps drop down menu.

Setup process


Log in to OSC OnDemand and request a Desktop/VDI session (this first screen shot below does not reflect the current, 2025, labelling in OnDemand).

Picture1.png

In a Desktop/VDI environment, open a terminal and run (this is a critical step; one cannot launch desmond jobs in maestro via the Schrodinger GUI in the Interactive Apps drop down menu.

module load desmond
maestro

In the main window of Maestro, you can open File and import structures or create new project

Screenshot 2022-06-29 120854.png

Screenshot 2022-06-29 121005.png

Once the structure is ready, navigate to the top right Tasks icon and find Desmond application; the details of this step depend on the software version; if you do not find desmond listed then use the search bar.

    Tasks >> Browse... > Applications tab >> Desmond

Screenshot 2022-06-29 121057 (2).png

In this example a Minimazation job will be done.

Screenshot 2022-06-29 121120.png

Make sure the Model system is ready:

  Model system >> Load from workspace >> Load

You can change the Job name; and you can write out the script and configuration files by clicking Write as shown below:

Screenshot 2022-06-29 121205.png

The green text will indicate the job path with the prefix "Job written to...". The path is a new folder located in the working directory indicated earlier.

Screenshot 2022-06-29 121315.png

Navigate using the terminal to that directory. You can modify the script to either run the simulation with a GPU or a CPU.

Run simulation with GPU

Navigate using the terminal to that directory and add the required SLURM directives and module commands at the top of the script, e.g.: desmond_min_job_1.sh:

#!/bin/bash
#SBATCH --time=0:20:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=1
#SBATCH --account=<account>

module reset
module load desmond/2023.2

# Desmond job script starts here

The setup is complete.


Run simulation with CPU only; this is no longer available, but is kept for posterity
Navigate using the terminal to that directory and edit the script, e.g.: desmond_min_job_1.sh:

"${SCHRODINGER}/utilities/multisim" -JOBNAME desmond_min_job_1 -HOST localhost -maxjob 1 -cpu 1 -m desmond_min_job_1.msj -c desmond_min_job_1.cfg -description Minimization desmond_min_job_1.cms -mode umbrella -set stage[1].set_family.md.jlaunch_opt=["-gpu"] -o desmond_min_job_1-out.cms -ATTACHED

Delete the -set stage[1].set_family.md.jlaunch_opt=["-gpu"] argument and change the -cpu argument from 1 to the number of CPUs you want, e.g. 8, resulting in

"${SCHRODINGER}/utilities/multisim" -JOBNAME desmond_min_job_1 -HOST localhost -maxjob 1 -cpu 8 -m desmond_min_job_1.msj -c desmond_min_job_1.cfg -description Minimization desmond_min_job_1.cms -mode umbrella  -o desmond_min_job_1-out.cms -ATTACHED

Add the required SLURM directives and module commands at the top of the script:

#!/bin/bash
#SBATCH --time=0:20:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH --account=<account>

module reset
module load desmond/2019.1

# Desmond job script starts here

The setup is complete.

 

Further Reading

Tag: 
Supercomputer: 
Service: 

FFTW

FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data. It is portable and performs well on a wide variety of platforms.

Availability and Restrictions

Versions

FFTW is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
3.3.10 X X X

You can use module spider fftw  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

FFTW is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

www.fftw.org, Open source

Usage

Usage

Set-up

Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module spider fftw to check what other modules need to be loaded first. Use  module load [module name and version] to load the necessary modules. Then use  module load fftw/3.3.10 to load the 3.3.10 FFTW module version.

Building with FFTW

The following environment variables are setup when the FFTW library is loaded:

VARIABLE USE
$FFTW3_CFLAGS Use during your compilation step for C programs.
$FFTW3_FFLAGS Use during your compilation step for Fortran programs.
$FFTW3_LIBS Use during your link step for the sequential version of the library.
$FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
$FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.

below is a set of example commands used to build a file called my-fftw.c .

module load fftw3
icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS

Further Reading

See Also

Supercomputer: 
Service: 

FSL

FSL is a library of tools for analyzing FMRI, MRI and DTI brain imaging data.

Availability and Restrictions

Versions

The following versions of FSL are available on OSC clusters:

Version Pitzer Ascend Cardinal
6.0.7.13 X X X*
* Curent default version

You can use module spider fsl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

FSL is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Analysis Group, University of Oxford/ freeware

Usage

Usage on Pitzer

Set-up

Configure your environment for use of FSL with module load fsl. This will load the default version.

Using FSL GUI

Access the FSL GUI with command for bash

source $FSLDIR/etc/fslconf/fsl.sh
fsl

For csh, one can use

source $FSLDIR/etc/fslconf/fsl.csh
fsl 

This will bring up a menu of all FSL tools. For information on individual FSL tools see FSL Overview page.

Using BASIL GUI

module load fsl/6.0.4
source $FSLDIR/etc/fslconf/fsl.sh
asl_gui --matplotlib

For more information, please visit https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/BASIL.

Further Reading 

Supercomputer: 
Fields of Science: 

FastQC

FastQC provides quality control checks of high throughput sequence data that identify areas of the data that may cause problems during further analysis.

Availability and Restrictions

Versions

FastQC is available on the Pitzer and Cardinal cluster. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
0.12.1 X X X*
* Current Default Version

You can use module spider fastqc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

FastQC is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Babraham Bioinformatics, Open source

Usage

Usage

Set-up

To configure your enviorment for use of FastQC, use command module load fastqc/0.12.1.

Further Reading

Supercomputer: 
Fields of Science: 

FreeSurfer

FreeSurfer is a software package used to anaylze nueroimaging data.

Availability & Restrictions

Versions

The following versions of FreeSurfer are available on OSC clusters:

Version Pitzer Ascend Cardinal Note
6.0.0   X X  
7.1.1        
7.2.0   X X  
7.3.0        
7.3.2 X X X  
7.4.1   X X*  
* Curent default version

You can use module spider freesurfer to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

FreeSurfer is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Athinoula A. Martinos Center, Open source

Usage

Usage on Pitzer

Set-up

Load the FreeSurfer module with  module load freesurfer. This will load the default version. Then, to continue configuring your environment, you must source the setup script for Freesurfer. Do this with the following command that corresponds to the Linux shell you are using. If using bash, use:

source $FREESURFER_HOME/SetUpFreeSurfer.sh

If using tcsh, use:

source $FREESURFER_HOME/SetUpFreeSurfer.csh

To finish configuring FreeSurfer, set the the FreeSurfer environment variable SUBJECTS_DIR to the directory of your subject data. The SUBJECTS_DIR variable defaults to the FREESURFER_HOME/subjects directory, so if this is your intended directory to use the enviornment set-up is complete.

To alter the SUBJECTS_DIR variable, however, again use the following command that corresponds to the Linux shell you are using. For bash:

export SUBJECTS_DIR=<path to subject data>

For tcsh:

setenv SUBJECTS_DIR=<path to subject data>

Note that you can set the SUBJECT_DIR variable before or after sourcing the setup script.

The cuda applications from FreeSurfer requires CUDA 5 library (which is not avaiable through module system). To set up cuda environment, run the following command after load the FreeSurfer module. If  you are using bash, run:

source $FREESURFER_HOME/bin/cuda5_setup.sh

If using tcsh, use:

source $FREESURFER_HOME/bin/cuda5_setup.csh

Further Reading 

Supercomputer: 
Service: 
Fields of Science: 

GAMESS

The General Atomic and Molecular Electronic Structure System (GAMESS) is a flexible ab initio electronic structure program. Its latest version can perform general valence bond, multiconfiguration self-consistent field, Möller-Plesset, coupled-cluster, and configuration interaction calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs well on open shell and excited state systems and can model relativistic effects. The GAMESS Home Page has additional information.

Availability and Restrictions

Versions

GAMESS is not currently available at the OSC.

VERSION

30 Sep 2019 (R2)
* Current default version

You can use module spider gamess to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

GAMESS is available to all OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Gordon research group, Iowa State Univ./ Proprietary freeware

Usage

Set-up

GAMESS usage is controlled via modules. Load one of the GAMESS modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

module load gamess  

Examples

Further Reading

General documentation is available from the GAMESS Home page and in the local machine directories.

Service: 

GATK

GATK is a software package for analysis of high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance.

Availability and Restrictions

Versions

The following versions of GATK are available on OSC clusters:

Version Pitzer Ascend Cardinal Notes
4.6.0.0 X X X*  
* Current default version

You can use module spider gatk to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

GATK4 is available to all OSC users under BSD 3-clause License.

GATK3 is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Broad Institute, Inc., BSD 3-clause License (GATK4 only)

Usage

Usage

Set-up

To configure your environment for use of GATK, run the following command: module load gatk/4.6.0.0

Usage

This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable. From module load gatk, a new environment variable, GATK, will be set. Thus, users can use the software by running the following command: gatk {other options},e.g  run gatk -h to see all options.

Known Issues

CBLAS undefined symbol error

Update: 05/22/2019 
Version: all

If you use GATK tools that need CBLAS (e.g. CreateReadCountPanelOfNormals), you might encounter an error as

INFO: successfully loaded /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so
java: symbol lookup error: /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dspr
java: symbol lookup error: /tmp/jniloader1239007313705592313netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dspr

The error raises because the system-default LAPACK does not support CBLAS.  The remedy is to run GATK in conjunction with lapack/3.8.0:

$ module load lapack/3.8.0
$ module load gatk/4.1.2.0
$ LD_LIBRARY_PATH=$OSC_LAPACK_DIR/lib64 gatk AnyTool toolArgs

Alternatively, we recommend using the GATK container. First, download the GATK container to your home or project directory

$ qsub -I -l nodes=1:ppn=1
$ cd $TMPDIR
$ export SINGULARITY_CACHEDIR=$TMPDIR
$ SINGULARITY_TMPDIR=$TMPDIR 
$ singularity pull docker://broadinstitute/gatk:4.1.2.0
$ cp gatk_4.1.2.0.sif ~/

Then run any GATK tool via

$ singularity exec ~/gatk_4.1.2.0.sif gatk AnyTool ToolArgs

You can read more about container in general from here. If you have any further questions, please contact OSC Help.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

GNU Compilers

Fortran, C and C++ compilers produced by the GNU Project. 

Availability and Restrictions

Versions

The GNU Compiler Collection (GCC) are available on all our clusters. These are the versions currently available:

Version Pitzer Ascend Cardinal Notes
11.4.1 X# X# X#  
12.3.0 X X X*  
13.2.0 X X X  
* Current Default Version
# System version
** There is always some version of the GNU compilers in the environment. If you want a specific version you should load the appropriate module. If you don't have a module loaded you will get either the system version or some other version, depending on what modules you do have loaded.

Modules

You can use module spider gcc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

To find out what version of gcc you are using, type gcc --version.

Access

The GNU compilers are available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

https://www.gnu.org/software/gcc/, Open source

Usage

Usage on Pitzer

Set-up

To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions): module load gcc/version.  For example, use module load gnu/8.1.0 to load GNU 8.1.0.

How to Compile

Once the module is loaded, follow the guides below for compile commands:

LANGUAGE NON-MPI MPI
Fortran 90 or 95 gfortran mpif90
Fortran 77 gfortran mpif77
c gcc mpicc
c++ g++ mpicxx

Building Options

The GNU compilers recognize the following command line options :

COMPILER OPTION PURPOSE
-fopenmp Enables compiler recognition of OpenMP directives (except mpif77)
-o FILENAME

Specifies the name of the object file

-O0 or no -O  option Disable optimization
-O1 or -O Ligh optimization
-O2 Heavy optimization
-O3 Most expensive optimization (Recommended)

 

 

 

 

 

 

There are numerous flags that can be used. For more information run man <compiler binary name>.

Known Issues

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

GROMACS

GROMACS is a versatile package of molecular dynamics simulation programs. It is primarily designed for biochemical molecules, but it has also been used on non-biological systems.  GROMACS generally scales well on OSC platforms. Starting with version 4.6 GROMACS includes GPU acceleration.

Availability and Restrictions

Versions

GROMACS is available on OSC Clusters. Both single and double precision executables are installed. The versions currently available at OSC are the following:

Version Pitzer Ascend Cardinal
2024.3 SPC(GNU); SP(Intel)   SPC(GNU); SP(Intel)
2024.4   SPC(GNU); SP(Intel)  
* Current default version; S = serial single node executables; P = parallel multinode; C = CUDA (GPU)

You should use module spider gromacs to view available modules for a given cluster. To select a particular software version, use module load gromacs/version. For example, use module load gromacs/2024.3 to load GROMACS version 2024.3; and after loading you should use module help gromacs/2024.3 to view details, such as, available executables (e.g., Intel builds do not have GPU executables), compiler prerequisites, additional modules required for specific executables, the suffixes of executables, etc.; some versions require specific prerequisite modules, and such details may be obtained with the command module spider gromacs/version.  Feel free to contact OSC Help if you need other versions for your work.

Access

GROMACS is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

https://www.gromacs.org/ Open source

Usage 

Usage on Pitzer

Set-up

To load the module for the default version of GROMACS, which initializes your environment for the GROMACS application, use module load gromacs/2024.3.

Using GROMACS

To execute a serial GROMACS versions 5 program interactively, simply run it on the command line, e.g.:

gmx pdb2gmx

Parallel multinode GROMACS versions 5 programs should be run in a batch environment with srun, e.g.:

srun gmx_mpi_d mdrun

Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision ('_gpu' denotes a GPU executable built with CUDA).  See the module help output for specific versions for more details on executable naming conventions.

Batch Usage

When you log into Pitzer you are actually connected to a login node. To  access the compute nodes, you must submit a job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session on Pitzer, one can run the following command:
sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00
which gives you one node and 40 cores (-N 1 -n 40) with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
Non-interactive Batch Job (Parallel Run)

batch script can be created and submitted for a serial, cuda (GPU), or parallel run. You can create the batch script using any text editor in a working directory on the system of your choice. Sample batch scripts and input files for all types of hardware resources are available here:

~srb/workshops/compchem/gromacs/

This simple batch script demonstrates some important points:

#!/bin/bash
# GROMACS Tutorial for Solvation Study of Spider Toxin Peptide
# see fwspider_tutor.pdf
#SBATCH --job-name fwsinvacuo.pitzer
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH --account=PZS0711
# turn off verbosity for noisy module commands
set +vx
module purge
module load intel/18.0.3
module load mvapich2/2.3
module load gromacs/2018.2
module list
set -vx

cd $SLURM_SUBMIT_DIR
echo $SLURM_SUBMIT_DIR
sbcast -p 1OMB.pdb $TMPDIR/1OMB.pdb
sbcast -p em.mdp $TMPDIR/em.mdp

cd $TMPDIR
mpiexec -ppn 1 gmx pdb2gmx -ignh -ff gromos43a1 -f 1OMB.pdb -o fws.gro -p fws.top -water none
mpiexec -ppn 1 gmx editconf -f fws.gro -d 0.7

mpiexec -ppn 1 gmx editconf -f out.gro -o fws_ctr.gro -center 2.0715 1.6745 1.914
mpiexec -ppn 1 gmx grompp -f em.mdp -c fws_ctr.gro -p fws.top -o fws_em.tpr -maxwarn 1
mpiexec -ppn 1 ls -l 
mpiexec gmx_mpi mdrun -s fws_em.tpr -o fws_em.trr -c fws_ctr.gro -g em.log -e em.edr

cp -p * $SLURM_SUBMIT_DIR/

* Note that sbcast does not recursively look through folders a loop in the jobscript is needed, please visit our Job Preparations page to learn more 

 

Further Reading

Supercomputer: 
Service: 

GSL

GSL is a library of mathematical methods for C and C++ languages.

Availability and Restrictions

Versions

GSL is available on all clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2.7.1 X X X*
* Current default version

You can use module spider gslto view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

GSL is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

GNU opensource

Usage

Set-up

To configure your environment for use of GSL, use the command module load gsl/version. For example, use module load gsl/2.7.1 to load version 2.7.1. 

Further Reading

Supercomputer: 

Gaussian

Gaussian is a very popular general purpose electronic structure program. Recent versions can perform density functional theory, Hartree-Fock, Möller-Plesset, coupled-cluster, and configuration interaction calculations among others. Geometry optimizations, vibrational frequencies, magnetic properties, and solution modeling are available. It performs well as black-box software on closed-shell ground state systems. 

Availability and Restrictions

Versions

Gaussian is available on the OSC Clusters. These versions are currently available at OSC (S means single node serial/parallel and C means CUDA, i.e., GPU enabled):

Version Pitzer Ascend Cardinal
g16c02 SC SC SC*
* Current default version; S = single node serial/parallel; C = CUDA, i.e., GPU enabled

You can use module spider gaussian to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of Gaussian for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

Publisher/Vendor/Repository and License Type

Gaussian, commercial

Usage

Usage on Pitzer

Set-up on Pitzer

To load the default version of the Gaussian module which initalizes your environment for Gaussian, use module load gaussian/g16c02

Using Gaussian

To execute Gaussian, simply run the Gaussian binary (g16 or g09) with the input file on the command line:

g16 < input.com

When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen with viewers or editors when the job is running in a batch queue because the batch output file, which captures standard output, is available in the directory from which the job was submitted.  Alternatively, Gaussian can be invoked without file redirection:

g16 input.com

in which case the output file will be named 'input.log' and its path will be the working directory when the command started; in this form outputs may not be available when the job is running in a batch queue, for example if the working directory was .

Batch Usage on Pitzer

When you log into pitzer.osc.edu you are logged into a login node. To gain access to the mutiple processors in the computing environment, you must submit your computations to the batch system for execution. Batch jobs can request mutiple processors and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session on Pitzer, one can run the following command:
sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00
which gives you 40 cores (-n 40) with 1 hour (-t 1:00:00). You may adjust the numbers per your need.
Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

/users/appl/srb/workshops/compchem/gaussian/

This simple batch script demonstrates the important points:

#!/bin/bash
#SBATCH --job-name=GaussianJob
#SBATCH --nodes=1 --ntasks-per-node=40
#SBATCH --time=1:00:00
#SBATCH --account=<project-account>

cp input.com $TMPDIR
# Use TMPDIR for best performance.
cd $TMPDIR
module load gaussian
g16 input.com
cp -p input.log *.chk $SLURM_SUBMIT_DIR

Running Gaussian jobs with GPU

Gaussian jobs can utilize the V100 GPUS of Pitzer.  GPUs are not helpful for small jobs but are effective for larger molecules when doing DFT energies, gradients, and frequencies (for both ground and excited states). They are also not used effectively by post-SCF calculations such as MP2 or CCSD. For more

The above example will utilize CPUs indexed from 0 to 19th, but 0th CPU is associated with 0th GPU.

A sample batch script for GPU on Pitzer is as follows:

#!/bin/tcsh
#SBATCH --job-name=methane
#SBATCH --output=methane.log
#SBATCH --nodes=1 --ntasks-per-node=48
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:00
#SBATCH --account=<project-account>

set echo
cd $TMPDIR
set INPUT=methane.com
# SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
cp $SLURM_SUBMIT_DIR/$INPUT .
module load gaussian/g16b01
g16 < ./$INPUT
ls -al
cp -p *.chk $SLURM_SUBMIT_DIR

 

A sample input file for GPU on Pitzer is as follows:

%nproc=48
%mem=8gb
%CPU=0-47
%GPUCPU=0=0
%chk=methane.chk
#b3lyp/6-31G(d) opt
methane B3LYP/6-31G(d) opt freq
0,1
C        0.000000        0.000000        0.000000
H        0.000000        0.000000        1.089000
H        1.026719        0.000000       -0.363000
H       -0.513360       -0.889165       -0.363000
H       -0.513360        0.889165       -0.363000

Known Issues

Out of Memory Problems for Large TMPDIR Jobs

For some Gaussian jobs, the operating system will start swapping and may trigger the out of memory (OOM) killer because of memory consumption by the local filesystem (TMPDIR) cache.  For these jobs %mem may not be critical, i.e., these jobs may not be big memory jobs per se; it is the disk usage that causes the OOM; known examples of this case are large ONIOM calculations.

While an investigation is ongoing, a simple workaround is to avoid putting the Gaussian internal files on TMPDIR.  The most obvious alternative to TMPDIR is PFSDIR, in which case the commands are

...
#SBATCH --gres=pfsdir
...
module load gaussian
export GAUSS_SCRDIR=$PFSDIR
...

 

Other workarounds exist; contact oschelp@osc.edu for details.

g16b01 G4 Problem

See the known issue and note that g16c01 is the current default module version.

Further Reading

 

Supercomputer: 
Service: 

Git

Git is a version control system used for tracking file changes and facilitating collaborative work.

Availability and Restrictions

Versions

Git is available on all OSC clusters. Only one version is available at any given time. To find out the current version:

git version

Access

Git is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Git, Open source

Usage

Set-up

No setup is required. You can use Apptainer/Singularity directly on all clusters.

Further Reading

Supercomputer: 

Gurobi

Gurobi is a mathematical optimization solver that supports a variety of programming and modeling languages.

Availability and Restrictions

Versions

The following versions of bedtools are available on OSC clusters:

Version Pitzer Ascend Cardinal
10.0.1     X*
12.0.0 X X X
* Current default version

You can use module spider gurobi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Gurobi is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

Publisher/Vendor/Repository and License Type

Gurobi Optimization, LLC/ Free academic floating license

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

HDF5

HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

Availability and Restrictions

Versions

HDF5 is available on the OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
1.14.3 X X X*
* Current Default Version

You can use module spider hdf5 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HDF5 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

The HDF Group, Open source (academic)

API Compatibility issue on hdf5/1.12

hdf5/1.12 may not compatible with applications created with earlier hdf5 versions. In order to work around, users may use a compatibility macro mapping:

  • To compile an application built with a version of HDF5 that includes deprecated symbols (the default), specify: -DH5_USE_110_API (autotools) or –DH5_USE_110_API:BOOL=ON (CMake)

However, users will not be able to take advantage of some of the new features in 1.12 if using these compatibility mappings. For more detail, please see release note.

Usage

Usage on Pitzer

Set-up

Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5/version. For example, run module load hdf5/1.14.3 to load version 1.14.3.

Building With HDF5

The HDF5 library provides the following variables for use at build time:

VARIABLE USE
$HDF5_C_INCLUDE Use during your compilation step for C programs
$HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
$HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
$HDF5_C_LIBS Use during your linking step programs
$HDF5_F90_LIBS

Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

icc -c $HDF5_C_INCLUDE myprog.c
icc -o myprog myprog.o $HDF5_C_LIBS
ifort -c $HDF5_F90_INCLUDE myprog.f90
ifort -o myprog myprog.o $HDF5_F90_LIBS

Batch Usage

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Non-interactive Batch Job (Serial Run)
batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
#!/bin/bash
#SBATCH --job-name=AppNameJob 
#SBATCH --nodes=1 --ntasks-per-node=48
#SBATCH --account <project-account>

module load hdf5/1.14.3
cp foo.dat $TMPDIR
cd $TMPDIR
appname
cp foo_out.h5 $SLURM_SUBMIT_DIR

Further Reading

Tag: 
Supercomputer: 
Service: 

HDF5-Serial

HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

For mpi-dependent codes, use the non-serial HDF5 module.

Availability and Restrictions

Versions

HDF5 is available for serial code on Pitzer Clusters. The versions currently available at OSC are:

Version Pitzer Notes
1.10.2 X  
1.10.4 X  
1.10.5 X  
1.12.0 X*  
1.12.2 X  
* Current Default Version

You can use module spider hdf5-serial to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HDF5 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

The HDF Group, Open source (academic)

Usage

Usage on Pitzer

Set-up

Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default serial HDF5 library, run the following command: module load hdf5-serial. To load a particular version, use module load hdf5-serial/version. For example, use module load hdf5-serial/1.10.5 to load HDF5 version 1.10.5. You can use module spider hdf5-serial to view available modules.

Building With HDF5

The HDF5 library provides the following variables for use at build time:

VARIABLE USE
$HDF5_C_INCLUDE Use during your compilation step for C programs
$HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
$HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
$HDF5_C_LIBS Use during your linking step programs
$HDF5_F90_LIBS

Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

icc -c $HDF5_C_INCLUDE myprog.c
icc -o myprog myprog.o $HDF5_C_LIBS
ifort -c $HDF5_F90_INCLUDE myprog.f90
ifort -o myprog myprog.o $HDF5_F90_LIBS

Batch Usage

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Non-interactive Batch Job (Serial Run)
batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
#PBS -N AppNameJob
#PBS -l nodes=1:ppn=28
module load hdf5
cd $PBS_O_WORKDIR
cp foo.dat $TMPDIR
cd $TMPDIR
appname
cp foo_out.h5 $PBS_O_WORKDIR

Further Reading

Supercomputer: 
Service: 

HISAT2

HISAT2 is a graph-based alignment program that maps DNA and RNA sequencing reads to a population of human genomes.

Availability and Restrictions

Versions

HISAT2 is available on the Pitzer Cluster. The versions currently available at OSC are:

Version Ascend
2.2.1 X

You can use module spider hisat2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HISAT2 is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

https://ccb.jhu.edu/software/hisat2, Open source

Usage

Usage

Set-up

To configure your enviorment for use of HISAT2, use command module load hisat2/2.2.1. This will load the version 2.2.1.

Further Reading

 
Supercomputer: 
Fields of Science: 

HPC Toolkit

HPC Toolkit is a collection of tools that measure a program's work, resource consumption, and inefficiency to analze performance.

Availability and Restrictions

Versions

The following versions of HPC Toolkitare available on OSC clusters:

Version Pitzer Ascend Cardinal
2023.08.1 X X X*
* Current default version

You can use module spider hpctoolkit to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HPC Toolkit is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Rice Univerity, Open source

Usage

Set-up

To configure your environment for use of HPC Toolkit, run the following command: module load hpctoolkit/version. For example, use module load hpctoolkit/2023.08.1 to load version 2023.08.1.

Further Reading

Supercomputer: 

HTSlib

HTSlib is a C library used for reading and writing high-throughput sequencing data. HTSlib is the core library used by SAMtools. HTSlib also provides the bgziphtsfile, and tabix utilities.

Availability and Restrictions

Versions

The versions of HTSlib currently available at OSC are:

Version Pitzer Ascend Cardinal
1.20 X X X*
* Current Default Version

You can use module spider htslib to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HTSlib is available to all OSC users.

Publisher/Vendor/Repository and License Type

Genome Research Ltd., Open source

Usage

Set-up

To configure your enviorment for use of HTSlib, use command module load htslib/version. For example, run module load htslib/1.20 to load version 1.20.

Further Reading

Supercomputer: 

Intel Compilers

The Intel compilers for both C/C++ and FORTRAN.

Availability and Restrictions

Old Intel compiler licenses for state-wide access with versions 19.1.3 and earlier are no longer available as of September 2, 2025. If you have any questions, please contact OSC Help.

Versions

The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Notes
2021.10.0 X X X* The last release of Intel Compiler Classic
* Current Default Version

You can use module spider intel  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

The Intel Compilers are available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Intel oneAPI Community License (For Academic use)

If you need the Intel compilers, tools, and libraries on your desktop or on your local clusters, Intel oneAPI is available without extra cost for most academic purposes: please read about Intel oneAPI.

Usage

Usage on Pitzer

Set-up on Pitzer

After you ssh to Pitzer, the default version of Intel compilers will be loaded for you automatically. 

Using the Intel Compilers

Once the intel compiler module has been loaded, the compilers are available for your use. See our compilation guide for suggestions on how to compile your software on our systems. The following table lists common compiler options available in all languages.

COMPILER OPTION PURPOSE
-c Compile only; do not link  
-DMACRO[=value] Defines preprocessor macro MACRO with optional value (default value is 1)  
-g  Enables debugging; disables optimization  
-I/directory/name Add /directory/name to the list of directories to be searched for #include files  
-L/directory/name Adds /directory/name to the list of directories to be searched for library files  
-lname Adds the library libname.a or libname.so to the list of libraries to be linked  
-o outfile Names the resulting executable outfile instead of a.out  
-UMACRO Removes definition of MACRO from preprocessor  
-v Emit version including gcc compatibility; see below
  Optimization Options
-O0 Disable optimization  
-O1 Light optimization  
-O2 Heavy optimization (default)  
-O3 Aggressive optimization; may change numerical results  
-ipo Inline function expansion for calls to procedures defined in separate files  
-funroll-loops Loop unrolling  
-parallel Automatic parallelization  
-openmp Enables translation of OpenMP directives  

The following table lists some options specific to C/C++

-strict-ansi Enforces strict ANSI C/C++ compliance
-ansi Enforces loose ANSI C/C++ compliance
-std=val Conform to a specific language standard

The following table lists some options specific to Fortran

-convert big_endian Use unformatted I/O compatible with Sun and SGI systems
-convert cray Use unformatted I/O compatible with Cray systems
-i8 Makes 8-byte INTEGERs the default
-module /dir/name Adds /dir/name to the list of directories searched for Fortran 90 modules
-r8 Makes 8-byte REALs the default
-fp-model strict Disables optimizations that can change the results of floating point calculations

Intel compilers use the GNU tools on the clusters:  header files, libraries, and linker.  This is called the Intel and GNU compatibility and interoperability.  Use the Intel compiler option -v to see the gcc version that is currently specified.  Most users will not have to change this.  However, the gcc version can be controlled by users in several ways. 

On OSC clusters the default mechanism of control is based on modules.  The most noticeable aspect of interoperability is that some parts of some C++ standards are available by default in various versions of the Intel compilers; other parts require an extra module.  The C++ standard can be specified with the Intel compiler option -std=val; see the compiler man page for valid values of val.

With an Intel 17 or 18 compiler, module cxx17 will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.   With an Intel 19 compiler, module gcc-compatibility will be automatically loaded by the intel module load command to enable the GNU tools necessary for the C++17 standard.  (In early 2020 OSC changed the name of these GNU tool controlling modules to clarify their purpose and because our underlying implementation changed.)

A symptom of broken gcc-compatibility is unusual or non sequitur compiler errors typically involving the C++ standard library especially with respect to template instantiation, for example:

    error: more than one instance of overloaded function "std::to_string" matches the argument list:
              detected during:
                instantiation of "..."

    error: class "std::vector<std::pair<short, short>, std::allocator<std::pair <short, short>>>" has no member "..."
              detected during:
                instantiation of "..."

An alternative way to control compatibility and interoperability is with Intel compiler options; see the "GNU gcc Interoperability" sections of the various Intel compiler man pages for details.

 

C++ Standard GNU Intel
C++11 > 4.8.1 > 14.0
C++14 > 6.1 > 17.0
C++17 > 7 > 19.0
C++2a features available since 8  

 

Batch Usage on Pitzer

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session

For an interactive batch session on Pitzer, one can run the following command:

sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00

which gives you 1 node (-N 1), 40 cores ( -n 40), and 1 hour ( -t 1:00:00). You may adjust the numbers per your need.

Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results . Below is the example batch script ( job.txt ) for a serial run:

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=40
#SBATCH --job-name hello
#SBATCH --account=<project-account>

module load intel
cp hello.c $TMPDIR
cd $TMPDIR
icc -O2 hello.c -o hello
./hello > hello_results
cp hello_results $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the   job.txt  file with the following command:

sbatch job.txt
Non-interactive Batch Job (Parallel Run)

Below is the example batch script ( job.txt ) for a parallel run:

#!/bin/bash
#SBATCH --time=1:00:00
#SBATCH --nodes=2 --ntasks-per-node=40
#SBATCH --job-name name
#SBATCH --account=<project-account>

module load intel
module laod intelmpi
mpicc -O2 hello.c -o hello
cp hello $TMPDIR
cd $TMPDIR
sun ./hello > hello_results
cp hello_results $SLURM_SUBMIT_DIR

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Intel MPI (Old)

Intel's implementation of the Message Passing Interface (MPI) library. See Intel Compilers for available compiler versions at OSC.

Availability and Restrictions

Versions

Intel MPI may be used as an alternative to - but not in conjunction with - the MVAPICH2 MPI libraries. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2017.4 X    
2018.3 X    
2018.4 X    
2019.3 X    
2019.7 X*    
2021.3 X    
2021.4.0   X*  
2021.5 X    
2021.10.0     X
2021.10 X X  
2021.11 X X  
* Current Default Version

You can use module spider intelmpi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Intel MPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Intel, Commercial

Usage

Usage on Pitzer

Set-up on Pitzer

To configure your environment for the default version of Intel MPI, use module load intelmpi.
Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi.

Using Intel MPI

Software compiled against this module will use the libraries at runtime.

Building With Intel MPI

On Oakley, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

VARIABLE USE
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
$MPI_LIBS Use when linking your program to Intel MPI.

In general, for any application already set up to use mpicc compilation should be fairly straightforward.

Batch Usage on Pitzer

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

Non-interactive Batch Job (Parallel Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Pitzer:
#!/bin/bash
#SBATCH --job-name MyIntelMPIJob
#SBATCH --nodes=2 --ntasks-per-node=48
#SBATCH --time=5:00:00
#SBATCH --account=<project-account>

module load intelmpi
srun my-impi-application

Usage on Ascend

Set-up on Ascend

To configure your environment for the default version of Intel MPI, use module spider intelmpi to check what module(s) to load first. Use module load [module name and version] to load what modules you need, then use module load intelmpi to load the default intelmpi.
Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi.

Using Intel MPI

Software compiled against this module will use the libraries at runtime.

Building With Intel MPI

On Oakley, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

VARIABLE USE
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
$MPI_LIBS Use when linking your program to Intel MPI.

In general, for any application already set up to use mpicc compilation should be fairly straightforward.

Batch Usage on Ascend

When you log into ascend.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

Non-interactive Batch Job (Parallel Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on Ascend:
#!/bin/bash
#SBATCH --job-name MyIntelMPIJob
#SBATCH --nodes=2 --ntasks-per-node=48
#SBATCH --time=5:00:00
#SBATCH --account=<project-account>

module load intelmpi
srun my-impi-application

Known Issues

A partial-node MPI job failed to start using mpiexec

Update: October 2020
Version: 2019.3 2019.7

A partial-node MPI job may fail to start using mpiexec from intelmpi/2019.3 and intelmpi/2019.7 with error messages like

[mpiexec@o0439.ten.osc.edu] wait_proxies_to_terminate (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:532): downstream from host o0439 was killed by signal 11 (Segmentation fault)
[mpiexec@o0439.ten.osc.edu] main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:2114): assert (exitcodes != NULL) failed
/var/spool/torque/mom_priv/jobs/11510761.pitzer-batch.ten.osc.edu.SC: line 30: 11728 Segmentation fault  
/var/spool/slurmd/job00884/slurm_script: line 24:  3180 Segmentation fault      (core dumped)

If you are using Slurm, make sure the job has CPU resource allocation using #SBATCH --ntasks=N instead of

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=N

If you are using PBS, please use Intel MPI 2018 or intelmpi/2019.3 with the module libfabric/1.8.1.

Using mpiexec/mpirun with Slurm

Update: October 2020
Version: 2017.x 2018.x 2019.x

Intel MPI on Slurm batch system is configured to support PMI process manager. It is recommended to use srun as MPI program launcher. If you prefer using mpiexec/mpirun over Hydra process manager with Slurm,  please add following code to the batch script before running any MPI executable:

unset I_MPI_PMI_LIBRARY I_MPI_HYDRA_BOOTSTRAP
export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0   # the option -ppn only works if you set this before

MPI-IO issues on home directories

Update: May 2020
Version: 2019.3
Certain MPI-IO operations with intelmpi/2019.3 may crash, fail or proceed with errors on the home directory. We do not expect the same issue on our GPFS file system, such as the project space and the scratch space. The problem might be related to the known issue reported by HDF5 group. Please read the section "Problem Reading A Collectively Written Dataset in Parallel" from HDF5 Known Issues for more detail.

Further Reading

See Also

Intel MPI

Intel's implementation of the Message Passing Interface (MPI) library. See Intel Compilers for available compiler versions at OSC.

Availability and Restrictions

Versions

Intel MPI may be used as an alternative to - but not in conjunction with - the MVAPICH MPI libraries. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2021.10.0 X X X*
2021.11.0     X
2021.12.1 X X X
2021.14.2     X
* Current Default Version

You can use module spider intel-oneapi-mpi to view available modules. Feel free to contact OSC Help if you need other versions for your work.

Note: The Intel Classic Compilers (icc, icpc, ifort) have been depreciated and are no longer included as of Intel oneAPI 2024.0. Earlier versions remain compatible with the Intel Classic Compilers, but Intel recommends switching to the Intel oneAPI compilers (icx, icpx, ifx).

Access

Intel MPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Intel, Commercial

Usage

Set-up

To configure your environment for the default version of Intel MPI, use module spider intel-oneapi-mpi to check what module(s) to load first. Use module load [module name and version] to load what modules you need, then use module load intel-oneapi-mpi/[version]to load intelmpi.

Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intel-oneapi-mpi.

Using Intel MPI

Software compiled against this module will use the libraries at runtime.

Building With Intel MPI

We have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

VARIABLE USE
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
$MPI_LIBS Use when linking your program to Intel MPI.

In general, for any application already set up to use mpicc compilation should be fairly straightforward.

Batch Usage

When you log into a cluster you are actually logged into a linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.

Non-interactive Batch Job (Parallel Run)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application) for five hours on a cluster:

#!/bin/bash
#SBATCH --job-name MyIntelMPIJob
#SBATCH --nodes=2 --ntasks-per-node=48
#SBATCH --time=5:00:00
#SBATCH --account=<project-account>

module load intel-oneapi-mpi/2021.10.0
srun my-impi-application

Known Issues

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Intel Math Kernel Library

Intel Math Kernel Library (MKL) consists of high-performance, multithreaded mathematics libraries for linear algebra, fast Fourier transforms, vector math, and more.

Availability and Restrictions

Versions

OSC supports single-process use of MKL for LAPACK and BLAS levels one through three. For multi-process applications, we also support the ScaLAPACK, FFTW2, and FFTW3 MKL wrappers. MKL modules are available for the Intel, GNU, and PGI compilers. MKL is available on Pitzer, Ascend and Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Notes
2023.2.0 X X X  
2024.1.0 X X X*  
2025.0.1     X  
*Current Default Version

You can use module spider intel-oneapi-mkl to view the available modules.

Feel free to contact OSC Help if you need other versions for your work.

Access

MKL is available to all OSC users.

Publisher/Vendor/Repository and License Type

Intel, Commercial

Usage

Set-up

To load the default MKL, run the following command: module load intel-oneapi-mkl/version. For example, run module load intel-oneapi-mkl/2024.1.0 to load version 2024.1.0.

This step is required for both building and running MKL applications. Note that loading an MKL module defines several environment variables that can be useful for compiling and linking to MKL, e.g., MKL_CFLAGS and MKL_LIBS

Intel Comipler

If you are using the MKL module with the Intel or oneAPI compiler, you may NOT need to load the MKL module separately. The Intel or oneAPI modules already include the MKLROOT variable, which allows most applications to automatically determine the required linking libraries and linker flags.

Intel MKL Advisor

Intel MKL provides multiple libraries to support various environments, tools, and interfaces. To determine the recommended libraries for a specific use case, use the Intel MKL Link Line Advisor  to obtain the appropriate linking methods and linker flags.

Exception: The "mkl" module is usually not needed when using the Intel compilers; just use the "-mkl" flag on the compile and link steps.

Dynamic Linking Variables

These variables indicate how to link to MKL. While their contents are used during compiling and linking, the variables themselves are usually specified during the configuration stage of software installation. The form of specification is dependent on the application software. For example, some softwares employing cmake for configuration might use this form:

cmake ..  -DMKL_INCLUDE_DIR="$MKLROOT/include"  -DMKL_LIBRARIES="MKL_LIBS_SEQ" 

Here is an exmple for some software employing autoconf:

./configure --prefix=$HOME/local/pkg/version CPPFLAGS="$MKL_CFLAGS" LIBS="$MKL_LIBS" LDFLAGS="$MKL_LIBS"

 

Variable Comment
MKL_LIBS Link with parallel threading layer of MKL
GNU_MKL_LIBS Dedicated for GNU compilers in Intel programming environment
MKL_LIBS_SEQ Link with sequential threading layer of MKL
GNU_MKL_LIBS_SEQ Dedicated for GNU compilers in Intel programming environment
MKL_SCALAPACK_LIBS Link with BLACS and ScaLAPACK of MKL
MKL_CLUSTER_LIBS Link with BLACS, CDFT and ScaLAPACK of MKL

Further Reading

Tag: 
Supercomputer: 
Service: 

Java

Java is a concurrent, class-based, object-oriented programming language.

Availability and Restrictions

Versions

Java is available on all OSC clusters. Only one version is available at any given time. To find out the current version, run:

java --version

Feel free to contact OSC Help if you need other versions for your work.

Access

Java is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Oracle, Freeware

Usage

Further Reading

Supercomputer: 

Julia

From julialang.org:

"Julia is a high-level, high-performance dynamic programming language for numerical computing. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing. In addition, the Julia developer community is contributing a number of external packages through Julia’s built-in package manager at a rapid pace. IJulia, a collaboration between the Jupyter and Julia communities, provides a powerful browser-based graphical notebook interface to Julia."

Availability and Restrictions

Versions

Julia is available on all the clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Notes
1.8.5 X      
1.10.4   X X*  
1.11.3     X  
*:Current default version

You can use module spider julia to view available modules for a given cluster. Feel free to contact OSC Help if you need other versions for your work.

Access

Julia is available for use by all OSC users.

Publisher/Vendor/Repository and License Type

Jeff Bezanson et al., Open source

Usage 

Interactive Julia Notebooks

If you are using OnDemand, you can simply work with Jupyter and the selection of the Julia kernel to use interactive notebooks to work on an OSC compute node!

Navigate to ondemand.osc.edu and select a Jupyter notebook:

Jupyter Notebook


Install Julia kernel for Jupyter

Since version 1.0, OSC users must manage their own IJulia kernels in Jupyter. The following is an example of adding the latest version of IJulia and creating the corresponding version of the Julia kernel:

$ module load julia/1.0.5
$ ~support/classroom/tools/create_julia_kernel
Installing IJulia
 Resolving package versions...
  Updating `~/.julia/environments/v1.0/Project.toml`
  [7073ff75] + IJulia v1.23.2
  Updating `~/.julia/environments/v1.0/Manifest.toml`
...
...
IJulia installed: 1.23.2
[ Info: Installing Julia kernelspec in /users/PAS1234/username/.local/share/jupyter/kernels/julia-1.0

In Juptyer Notebook, you can find the item Julia 1.0.5 in the kernel list:

Screen Shot 2021-08-19 at 12.46.48 AM.png

For more detail about package management, please refer to the Julia document

Acess gurobi from Jupyter Notebook

 To acess gurobi from Jupyter notebook, users would need to request access for the Gurobi software. More information can be found at Gurobi webpage. User would need to set the path to the gurobi license file located on a generic cluster in the notebook as follows,

ENV["GRB_LICENSE_FILE"] = "/usr/local/gurobi/10.0.1/gurobi.lic"

 

 

 
Supercomputer: 
Service: 
Fields of Science: 

LAMMPS

The Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a classical molecular dynamics code designed for high-performance simulation of large atomistic systems.  LAMMPS generally scales well on OSC platforms, provides a variety of modeling techniques, and offers GPU accelerated computation.

Availability and Restrictions

Versions

LAMMPS is available on all clusters. The following versions are currently installed at OSC:

Version Pitzer Ascend Cardinal
20230802.3 PC PC PC*
20250722 PC PC PC
* Current default version; S = serial executables; P = parallel; C = CUDA
*  IMPORTANT NOTE: You must load the correct compiler and MPI modules before you can load LAMMPS. To determine which modules you need, use module spider lammps/{version}.  Some LAMMPS versions may be available with multiple compiler versions and MPI versions; in general, we recommend using the latest versions of tools and application software, but users should benchmark a production ready simulation to gauge performance.

You can use module spider lammps  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

LAMMPS is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Sandia National Lab., Open source

Usage

Usage

Set-up

To load a version of LAMMPS module and set up your environment, use  module load lammps/version. For example, use module load lammps/20230802.3to load version 20230802.3.

Using LAMMPS

Once a module is loaded, LAMMPS can be run interactively, to obtain info or to simulate a tiny system, with the following command:
lmp < input.file

Note that the lammps wrapper script, called lammps, is no longer provided by OSC.  To see information on the packages and executables for a particular installation, run the module help command, for example:

module help lammps

Batch Usage

To access a cluster's main computational resources, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session

For an interactive batch session one can run the following command:

sinteractive -A <project-account> -N 1 -n 48 -g 1 -t 00:20:00 

which requests one whole node with 28 cores ( -N 1 -n 48), for a walltime of 20 minutes ( -t 00:20:00 ), with one gpu (-g 1). You may adjust the numbers per your need.

Non-interactive Batch Job (Parallel Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

~srb/workshops/compchem/lammps/

Below is a sample batch script. It asks for 56 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

#!/bin/bash
#SBATCH --job-name=chain 
#SBATCH --nodes=2 --ntasks-per-node=48 
#SBATCH --time=10:00:00 
#SBATCH --account=<project-account>

module load lammps/20230802.3
sbcast -p chain.in $TMPDIR/chain.in
cd $TMPDIR 
srun lmp < chain.in 
sgather -pr $TMPDIR $SLURM_SUBMIT_DIR/output

Further Reading

Supercomputer: 
Service: 

LAPACK

LAPACK (Linear Algebra PACKage) provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.

Availability and Restrictions

A highly optimized implementation of LAPACK is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building LAPACK for yourself. MKL is available to all OSC users.

Publisher/Vendor/Repository and License Type

http://www.netlib.org/lapack/, Open source

Usage

See OSC's MKL software page for usage information. Note that there are lapack shared libraries on the clusters; however, these are old versions from the operating system and should generally not be used.  You should modify your makefile or build script to link to the MKL libraries instead; a quick start for a crude approach is to merely load an mkl module and substitute the consequently defined environment variable $(MKL_LIBS) for -llapack.

Further Reading

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

LS-DYNA

LS-DYNA is a general purpose finite element code for simulating complex structural problems, specializing in nonlinear, transient dynamic problems using explicit integration. LS-DYNA is one of the codes developed at Livermore Software Technology Corporation (LSTC).

Availability and Restrictions

Versions

LS-DYNA is available on Cardinal Cluster for both serial (smp solver for single node jobs) and parallel (mpp solver for multipe node jobs) versions. The versions currently available at OSC are:

Version Cardinal
11.2.2 mpp X
13.1.0 smp X
mpp X
15.0.2 smp X
mpp X
* Current default version

You can use module spider ls-dyna to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

ls-dyna is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

Access for Commercial Users

Contact OSC Help for getting access to LS-DYNA if you are a commercial user.

Publisher/Vendor/Repository and License Type

LSTC, Commercial

Usage

Usage on Cardinal

Set-up on Cardinal

To view available modules installed on Cardinal, use  module spider ls-dyna for smp solvers, and use  module spider mpp for mpp solvers. In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp-dyna/971_d_9.0.1 is the mpp solver with double precision on Cardinal. Use  module load name to load LS-DYNA with a particular software version. For example, use  module load mpp-dyna/971_d_9.0.1 to load LS-DYNA mpp solver version 9.0.1 with double precision on Cardinal.

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

For an interactive batch session one can run the following command:

sinteractive -A <project-account> -N 1 -n 48 -t 00:20:00 -L lsdyna@osc:48
which requests one whole node with 58 cores (-N 1 -n 48), for a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.
Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Please follow the steps below to use LS-DYNA via the batch system:

1) copy your input files (explorer.k in the example below) to your work directory at OSC

2) create a batch script, similar to the following file, saved as job.txt. It uses the smp solver for a serial job (nodes=1) on Cardinal:

#!/bin/bash
#SBATCH --job-name=plate_test
#SBATCH --time=5:00:00
#SBATCH --nodes=1 --ntasks-per-node=48
#SBATCH --account <project-account>
#SBATCH -L lsdyna@osc:48

# The following lines set up the LSDYNA environment
module load ls-dyna/971_d_9.0.1
#
# Run LSDYNA (number of cpus > 1)
#

lsdyna I=explorer.k NCPU=48 

 3) submit the script to the batch queue with the command: sbatch job.txt.

 When the job is finished, all the result files will be found in the directory where you submitted your job ($SLURM_SUBMIT_DIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. 

Non-interactive Batch Job (Parallel Run)
Please follow the steps below to use LS-DYNA via the batch system:

1) copy your input files (explorer.k in the example below) to your work directory at OSC

2) create a batch script, similar to the following file, saved as job.txt). It uses the mmp solver for a parallel job (nodes>1) on Cardinal:

#!/bin/bash
#SBATCH --job-name=plate_test 
#SBATCH --time=5:00:00 
#SBATCH --nodes=2 --ntasks-per-node=48 
#SBATCH --account <project-account>
#SBATCH -L lsdyna@osc:56

# The following lines set up the LSDYNA environment
module load intel/18.0.3
module load intelmpi/2018.3
module load mpp-dyna/971_d_9.0.1

#
# Run LSDYNA (number of cpus > 1)
#
srun mpp971 I=explorer.k NCPU=56 

 3) submit the script to the batch queue with the command: sbatch job.txt.

When the job is finished, all the result files will be found in the directory where you submitted your job ($SLURM_SUBMIT_DIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

...
cd $TMPDIR
sbcast $SLURM_SUBMIT_DIR/explorer.k explorer
... #launch the solver and execute
sgather -pr $TMPDIR ${SLURM_SUBMIT_DIR}
#or you may specify a directory for your output files, such as
#sgather -pr $TMPDIR ${SLURM_SUBMIT_DIR}/output

Further Reading

See Also

Supercomputer: 
Service: 

LS-OPT

LS-OPT is a package for design optimization, system identification, and probabilistic analysis with an interface to LS-DYNA.

Availability and Restrictions

Versions

The following versions of ls-opt are available on OSC clusters:

Version Cardinal
6.0.0 X*
* Current default version

You can use module spider ls-opt to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

In order to use LS-OPT, you need LS-DYNA. ls-dyna is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

Publisher/Vendor/Repository and License Type

LSTC, Commercial

Usage

Usage on Cardinal

Set-up

To configure your environment for use of LS-OPT, run the following command: module load ls-opt. The default version will be loaded. To select a particular LS-OPT version, use module load ls-opt/version. For example, use module load ls-opt/6.0.0 to load LS-OPT 6.0.0.

Further Reading

Supercomputer: 
Service: 

LS-PrePost

LS-PrePost is an ad­vanced pre and post-proces­sor that is de­liv­ered free with LS-DY­NA.

Availability and Restrictions

Versions

The following versions of ls-prepost are available on OSC clusters:

Version Cardinal
4.6 X*
* Current default version

You can use module spider ls-prepost to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

In order to use LS-PrePost you need LS-DYNA. ls-dyna is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

Publisher/Vendor/Repository and License Type

LSTC, Commercial

Usage

Usage on Cardinal

Set-up

To configure your environment for use of LS-PrePost, run the following command: module load ls-prepost. The default version will be loaded. To select a particular LS-PrePost version, use module load ls-prepost/<version>. For example, use module load ls-prepost/4.6 to load LS-PrePost 4.6.

Further Reading

Supercomputer: 
Service: 

User-Defined Material for LS-DYNA

This page describes how to specify user defined material to use within LS-DYNA.  The user-defined subroutines in LS-DYNA allow the program to be customized for particular applications.  In order to define user material, LS-DYNA must be recompiled.

Usage

The first step to running a simulation with user defined material is to build a new executable. The following is an example done with solver version mpp971_s_R7.1.1.

When you log into the Oakley system, load mpp971_s_R7.1.1 with the command:

module load mpp-dyna/R7.1.1

Next, copy the mpp971_s_R7.1.1 object files and Makefile to your current directory:

cp /usr/local/lstc/mpp-dyna/R7.1.1/usermat/* $PWD

Next, update the dyn21.f file with your user defined material model subroutine. Please see the LS-DYNA User's Manual (Keyword version) for details regarding the format and structure of this file.

Once your user defined model is setup correctly in dyn21.f, build the new mpp971 executable with the command:

make

To execute a multi processor (ppn > 1) run with your new executable, execute the following steps:

1) move your input file to a directory on an OSC system (pipe.k in the example below)

2) copy your newly created mpp971 executable to this directory as well

3) create a batch script (lstc_umat.job) like the following:

#PBS -N LSDYNA_umat
#PBS -l walltime=1:00:00
#PBS -l nodes=2:ppn=8
#PBS -j oe
#PBS -S /bin/csh

# This is the template batch script for running a pre-compiled
# MPP 971 v7600 LS-DYNA.  
# Total number of processors is ( nodes x ppn )
#
# The following lines set up the LSDYNA environment
module load mpp-dyna/R7.1.1
#
# Move to the directory where the job was submitted from
# (i.e. PBS_O_WORKDIR = directory where you typed qsub)
#
cd $PBS_O_WORKDIR
#
# Run LSDYNA 
# NOTE: you have to put in your input file name
#
mpiexec mpp971 I=pipe.k NCPU=16

          4) Next, submit this job to the batch queue with the command:

       qsub lstc_umat.job

The output result files will be saved to the directory you ran the qsub command from (known as the $PBS_O_WORKDIR_

Documentation

On-line documentation is available on LSTC website.

See Also

 

 

Service: 

Linaro HPC tools

Linaro HPC tools analyze how HPC software runs. It consists of three applications, Linaro DDT, Linaro Performance Reports and Linaro MAP: 

  • Linaro DDT: graphical debugger for HPC applications.
  • Linaro MAP: HPC application profiler with easy-to-use GUI environment.
  • Linaro Performance Reports: simple tool to generate a single-page HTML or plain text report that presents overall performance characteristics of HPC applications.
NOTE: Because Linaro has aquired ARM's forge products, all ARM module files have been renamed accordingly. Allinear/ARM modules are still available and have same functionality as new Linaro modules.
NOTE: Because ARM has aquired Allinea, all Allinea module files have been renamed accordingly. Allinea modules are still available and have same functionality as new ARM modules.
NOTE [June 29, 2022]: As ARM reported security vulnerabilities on the old ARM Forge versions prior to 22.0.x, we have removed the old versions and installed 22.0.2 version.

Availability & Restrictions

Versions

The following versions of Linaro HPC tools are available on OSC clusters:

Version Cardinal
23.0.4 X
* Current default version

You can use module spider linaro-forge to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Linaro DDT, MAP and Performance Reports are available to all OSC users.

Publisher/Vendor/Repository and License Type

Linaro, Commercial

Usage

Linaro DDT

Linaro DDT is a debugger for HPC software that automatically alerts users of memory bugs and divergent behavior. For more features and benefits, visit Linaro Forge - Linaro DDT.

For usage instructions and more iformation, read Linaro DDT.

Linaro MAP

Linaro MAP produces a detailed profile of HPC software. Unlike Linaro Performance Reports, you must have the source code to run Linaro MAP because its analysis details the software line-by-line. For more features and benefits, visit Linaro Forge - Linaro MAP

For usage instructions and more information, read Linaro MAP.

Linaro Performance Reports

Linaro Performance Reports analyzes and documents information on CPU, MPI, I/O, and Memory performance characteristics of HPC software, even third party code, to aid understanding about the overall performance. Although it should not be used all the time, Linaro Performance Reports is recommended to OSC users as a viable option to analyze how an HPC application runs. View an example report to navigate the format of a typical report. For more example reports, features and benefits, visit Linaro Forge - Linaro Performance Reports.

For usage instructions and more information, read Linaro Performance Reports.

Troubleshooting

Using Linaro Forge software with MVAPICH2

As noted in Linaro's User Guide:

Some MPIs, most notably MVAPICH, are not yet supported by Express Launch mode
(in which you can just put “perf-report” in front of an existing mpirun/mpiexec line). These can
still be measured using the Compatibility Launch mode.

Instead of this Express Launch command:

perf-report mpiexec <mpi args> <program> <program args> # BAD

Use the compatibility launch version instead:

perf-report -n <num procs> --mpiargs="<mpi args>" <program> <program args>

Further Reading

See Also

Documentation Attachment: 
Supercomputer: 
Service: 
Fields of Science: 

Linaro Performance Reports

Linaro Performance Reports is a simple tool used to generate a single-page HTML or plain text report that presents the overall performance characteristics of HPC applications. It supports pthreads, OpenMP, or MPI code on CPU, GPU, and MIC based architectures.

Availability and Restrictions

Versions

The versions currently available at OSC are:

Version Pitzer Ascend
22.0.2 X X
23.1 X* X*
* Current default version

You can use module spider linaro-pr to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Linaro Performance Reports is available to all OSC users. We have 64 seats with 64 HPC tokens. Users can monitor the license status here.

Publisher/Vendor and License Type

Linaro, Commercial

Usage

Set-up

To load the module for the Linaro Performance Reports default version, use module load linaro-pr. To select a particular software version, use module load linaro-pr/version. For example, use module load linaro-pr/6.0 to load Linaro Performance Reports version 6.0, provided the version is available on the OSC cluster in use.

Using Linaro Performance Reports

You can use your regular executables to generate performance reports. The program can be used to analyze third-party code as well as code you develop yourself. Performance reports are normally generated in a batch job.

To generate a performance report for an MPI program:

module load linaro-pr
perf-report -np <num procs> --mpiargs="<mpi args>" <program> <program args>

where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

perf-report -np 12 wave_c

To generate a performance report for a non-MPI program:

module load linaro-pr
perf-report --no-mpi <program> <program args>

The performance report is created in both html and plain text formats. The file names are based on the executable name, number of processes, date and time, for example,  wave_c_12p_2016-02-05_12-46.html. To open the report in html format use

firefox wave_c_12p_2016-02-05_12-46.html

For more details, download the Linaro Performance Reports User Guide.

Performance Reports with GPU

Linaro Performance Reports can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Linaro Performance Reports with

perf-report {executable}

For more information, please read the section 6.10 of the Linaro Performance Reports User Guide.

Further Reading

See Also

Documentation Attachment: 
Supercomputer: 
Service: 

Linaro MAP

Linaro MAP is a full scale profiler for HPC programs. We recommend using Linaro MAP after reviewing reports from Linaro Performance Reports. MAP supports pthreads, OpenMP, and MPI software on CPU, GPU, and MIC based architectures.

Availability & Restrictions

Versions

The Linaro MAP versions currently available at OSC are:

Version Pitzer Ascend
22.0.2 X X
23.1 X* X*
* Current default version

You can use module spider linaro-forge to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Linaro MAP is available to all OSC users. We have 64 seats with 80 HPC tokens. Users can monitor the Linaro License Server Status.

Publisher/Vendor and License Type

Linaro, Commercial

Usage

Set-up

To load the default version of the Linaro MAP module, use module load linaro-forge. To select a particular software version, use module load linaro-forge/version. For example, use module load linaro-forge/6.0 to load Linaro MAP version 6.0, provided the version is available on the cluster in use. 

Note: Before you run MAP from the command line for the first time, open MAP as a GUI from OnDemand to configure with appropriate settings for your environment.

Using Linaro MAP

Profiling HPC software with Linaro MAP typically involves three steps: 

1. Prepare the executable for profiling.

Regular executables can be profiled with Linaro MAP, but source code line detail will not be available. You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other appropriate compiler options. For example:

mpicc wave.c -o wave -g -O3

This executable built with the debug flag can be used for Linaro Performance Reports as well.

Note: The -g flag turns off all optimizations by default. For profiling your code you should use the same optimizations as your regular executable, so explicitly include the -On flag, where n is your normal level of optimization, typically -O2 or -O3, as well as any other compiler optimization options.

2. Run your code to produce the profile data file (.map file).

Profiles are normally generated in a batch job.  To generate a MAP profile for an MPI program:

module load linaro-forge
map --profile -np <num proc> --mpiargs="<mpi args>" <program> <program args>

where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to srun (other than -n), <program> is the executable to be run and <program args> represents arguments passed to your program.

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

map --profile -np 12 wave_c

To profile a non-MPI program:

module load linaro-forge
map --profile --no-mpi <program> <program args>

The profile data is saved in a .map file in your current directory.

As a result of this step, a .map file that is the profile data file is created in your current directory. The file name is based on the executable name, number of processes, date and time, for example, wave_c_12p_2016-02-05_12-46.map.

For more details on using Linaro MAP, refer to the Linaro Forge User Guide.

3. Analyze the profile data file using either the Linaro local client or the MAP GUI.

You can open the profile data file using a client running on your local desktop computer. For client installation and usage instructions, please refer to the section: Client Download and Setup. This option typically offers the best performance.

Alternatively, you can run MAP in interactive mode, which launches the graphical user interface (GUI).  For example:

map wave_c_12p_2016-02-05_12-46.map

For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

MAP with GPU

Linaro MAP can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Linaro MAP with

map {executable}

For more information, please read the Chapter 15 of the Linaro Forge User Guide.

Client Download and Setup

1. Download the client.

To download the client, go to the Linaro website and choose the appropriate Linaro Forge remote client download for Windows, Mac, or Linux. For Windows and Mac, just double click on the downloaded file and allow the installer to run. For Linux, extract the tar file using the command tar -xf file_name and run the installer in the extracted file directory with ./installer. Please contact OSC Help, if you have any issues on downloading the client.

2. Configure the client.

After installation, you can configure the client as follows:

  • Open the client program. For Windows or Mac, just click the desktop icon or navigate to the application through its file path. For Linux use the command {linaro-forge-path}/bin/map.

  • Once the program is launched, select Linaro MAP in the left column.
  • In the Remote Launch drop down menu, select "Configure...".
  • Click Add to create a new profile for your login.
  • In the Host Name section, type your ssh connection. For example: "username@ruby.osc.edu".
  • For Remote Installation Directory, type /usr/local/linaro/forge-{version}, specifying the Linaro Forge version number that created the data profile file you are attempting to view. For example, /usr/local/linaro/forge-7.0 for Linaro Forge version 7.0.
  • You can test your login information by clicking Test Remote Launch. It will ask your password. Use the same password for the cluster login.
  • Close the Configure window. You will see a new option under the Remote Launch drop down menu for the host name you entered. Select your profile and login with your password. 
  • If the login was successful, then you should see License Serial:XXX in the bottom left corner of the window.

This login configuration is needed only for the first time of use. In subsequent times, you can just select your profile.

3. Open the profile data file.

After login, click on LOAD PROFILE DATA FILE. This opens a file browser of your home directory on the OSC cluster you logged onto. Go to the directory that contains the .map file and select it. This will open the file and allow you to navigate the source code line-by-line and investigate the performance characteristics. 

A license is not required to simply open the client, so it is possible to skip 2. Configure the client, if you download the profile data file to your desktop. You can then open it by just selecting LOAD PROFILE DATA FILE and navigating through a file browser on your local system.

Further Reading

See Also

Documentation Attachment: 
Supercomputer: 
Service: 
Fields of Science: 

Linaro DDT

Linaro DDT is a graphical debugger for HPC applications. It supports pthreads, OpenMP, or MPI code on CPU, GPU, and MIC based architectures.

Availability & Restrictions

Versions

The Linaro DDT versions currently available at OSC are:

Version Pitzer Ascend
22.0.2 X X
23.1 X* X*
* Current default version

You can use module spider linaro-ddt to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Linaro DDT is available to all OSC users. We have 64 seats with 80 HPC tokens. Users can monitor the Linaro License Server Status.

Publisher/Vendor and License Type

Linaro, Commercial

Usage

Set-up

To load the module for the Linaro DDT default version, use module load linaro-ddt. To select a particular software version, use module load linaro-ddt/version. For example, use module load linaro-ddt/23.1 to load Linaro DDT version 23.1, provided the version is available on the OSC cluster in use.

Note: Before you run DDT from the command line for the first time, open DDT as a GUI from OnDemand to configure with appropriate settings for your environment.

Using Linaro DDT

DDT debugs executables to generate DDT reports. The program can be used to debug third-party code as well as code you develop yourself. DDT reports are normally generated in a batch job.

To generate a DDT report for an MPI program:

module load linaro-ddt
ddt --offline -np <num procs> --mpiargs="<mpi args>" <program> <program args>

where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

ddt --offline -np 12 wave_c

To debug a non-MPI program:

module load linaro-ddt
ddt --offline --no-mpi <program> <program args>

The DDT report is created in html format. The file names are based on the executable name, number of processes, date and time, for example, wave_c_12p_2016-02-05_12-46.html. To open the report use

firefox wave_c_12p_2016-02-05_12-46.html

Using the Linaro DDT GUI

To debug with the DDT GUI remove the --offline option. For example, to debug the MPI program above, use

ddt -np 12 wave_c

For a non-MPI program:

ddt --no-mpi <program> <program args>

This will open the DDT GUI, enabling interactive debugging options.

For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

For more details, see the Linaro DDT developer page.

DDT with GPU

DDT can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Linaro Performance Reports with

ddt {executable}

For more information, please read the chapter 14 of the Linaro Forge User Guide.

Supercomputer: 

MATLAB

MATLAB is a technical computing environment for high-performance numeric computation and visualization. MATLAB integrates numerical analysis, matrix computation, signal processing, and graphics in an easy-to-use environment where problems and solutions are expressed just as they are written mathematically--without traditional programming.

Availability and Restrictions

Versions

MATLAB is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
r2024a X X X*
r2024b     X
* Current default version

You can use module spider matlab to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Matlab is also available through Jupyter, please see the Running MATLAB on Jupyter section for more information.

Access: Academic Users Only (non-commercial, non-government)

Academic users can use Matlab at OSC. All users must be added to the license server before using MATLAB. Please contact OSC Help to be granted access or for any license related questions.

Publisher/Vendor/Repository and License Type

MathWorks, Commercial (University site license)

Toolboxes and Features

OSC's current licenses support the following MATLAB toolboxes and features (please contact OSC Help for license-specific questions):

MATLAB
Simulink
5G Toolbox
AUTOSAR Blockset
Aerospace Blockset
Aerospace Toolbox
Antenna Toolbox
Audio Toolbox
Automated Driving Toolbox
Bioinformatics Toolbox
Communications Toolbox
Computer Vision Toolbox
Control System Toolbox
Curve Fitting Toolbox
DDS Blockset
DSP System Toolbox
Data Acquisition Toolbox
Database Toolbox
Datafeed Toolbox
Deep Learning HDL Toolbox
Deep Learning Toolbox
Econometrics Toolbox
Embedded Coder
Filter Design HDL Coder
Financial Instruments Toolbox
Financial Toolbox
Fixed-Point Designer
Fuzzy Logic Toolbox
GPU Coder
Global Optimization Toolbox
HDL Coder
HDL Verifier
Image Acquisition Toolbox
Image Processing Toolbox
Instrument Control Toolbox
LTE Toolbox
Lidar Toolbox
MATLAB Coder
MATLAB Compiler SDK
MATLAB Compiler
MATLAB Report Generator
Mapping Toolbox
Mixed-Signal Blockset
Model Predictive Control Toolbox
Model-Based Calibration Toolbox
Motor Control Blockset
Navigation Toolbox
OPC Toolbox
Optimization Toolbox
Parallel Computing Toolbox
Partial Differential Equation Toolbox
Phased Array System Toolbox
Powertrain Blockset
Predictive Maintenance Toolbox
RF Blockset
RF PCB Toolbox
RF Toolbox
ROS Toolbox
Radar Toolbox
Reinforcement Learning Toolbox
Risk Management Toolbox
Robotics System Toolbox
Robust Control Toolbox
Satellite Communications Toolbox
Sensor Fusion and Tracking Toolbox
SerDes Toolbox
Signal Integrity Toolbox
Signal Processing Toolbox
SimBiology
SimEvents
Simscape Driveline
Simscape Electrical
Simscape Fluids
Simscape Multibody
Simscape
Simulink 3D Animation
Simulink Check
Simulink Code Inspector
Simulink Coder
Simulink Compiler
Simulink Control Design
Simulink Coverage
Simulink Design Optimization
Simulink Design Verifier
Simulink Desktop Real-Time
Simulink PLC Coder
Simulink Real-Time
Simulink Report Generator
Simulink Requirements
Simulink Test
SoC Blockset
Spreadsheet Link
Stateflow
Statistics and Machine Learning Toolbox
Symbolic Math Toolbox
System Composer
System Identification Toolbox
Text Analytics Toolbox
UAV Toolbox
Vehicle Dynamics Blockset
Vehicle Network Toolbox
Vision HDL Toolbox
WLAN Toolbox
Wavelet Toolbox
Wireless HDL Toolbox

See this page if you need to install additional toolbox by yourself. 

Usage

Usage on Pitzer

Set-up

To load the default version of MATLAB module, use module load matlab.

Running MATLAB

The following command will start an interactive, command line version of MATLAB:

matlab -nodisplay 
If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab. For more information about the matlab command usage, type  matlab –h for a complete list of command line options.

The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

Batch Usage

When you log into pitzer.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Interactive Batch Session
For an interactive batch session using the command line version of MATLAB, one can run the following command:
sinteractive -A <project-account> -N 1 -n 40 -t 00:20:00

which requests one whole node with 40 cores ( -N 1 -n 40), for a walltime of 20 minutes ( -t 00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above. You may adjust the numbers per your need.

Additional Topics

MATLAB Parallel Functions and Tools

MATLAB now supports Parallel Computing Toolbox. The Parallel Computing Toolbox lets you solve computationally and data-intensive programs using multiple cores and GPUs. Built in MATLAB functions and tools allow for easy parallelization of MATLAB applications. Programs can be run both interactively or as batch jobs.

Currently only r2019b and newer versions have full support for the Parallel Computing Toolbox on Pitzer. 

Please refer to the official MATLAB documentation for more information on the Parallel Computing Toolbox

Sections:

Creating Parallel Pools

You can parallelize by requesting a certain number of workers and then work can be offloaded onto those pool of workers. For local computations, the number of workers you can requests relates to the number of cores available.


To start up a pool you can run:

p = gcp

p is the pool object which can be used to check information on the worker pool.

By default gcp creates a pool of workers equal to the number of cores on the job.

Note:

  • It may takes a couple of seconds to a minute to start up a pool.
  • You cannot run multiple parallel pools at the same time on a single job.


To delete the current pool if one exists run:

delete(gcp('nocreate')

After the program is done running the pool will still remain active. MATLAB only deletes the pool after the default 30 minutes. So if you want to end a pool you must manually delete it, let MATLAB timeout the pool. or terminate the job. If you make changes to the code interactively it is recommended you delete the pool and spin up a new pool of workers.


See Matlab documentation for more information on worker pools  here

 

Parpool and Batch

Parallel jobs can also be submitted by a Matlab script, as is demonstrated below in the Submitting Single-Node Parallel MATLAB Jobs and Submitting Multi-Node Parallel MATLAB Jobs sections. The 2 main ways of doing so is through parpool and batch.

First, before using parpool or batch, you must get a handle to the profile cluster. To do this use the parcluster function.

% creates cluster profile object for the specified cluster profile
c = parcluster("Cluster_Profile");
% creates a cluster object to your current job
c = parcluster("local");

See the Submitting Multi-Node Parallel MATLAB Jobs section below for more information on how to create a cluster profile.

Creating the object with a cluster profile will result in a new job to be submitted when launching parpool or batch. Make sure the appropriate arguments are set in the cluster profile. Creating the object with the 'local' parameter will not result in a new job to be launched when executing parpool or batch. Instead the workers will be allocated to the cores in your current job. 

Once you have you a profile object created, you can now launch parallel jobs.

To launch a parpool parallel job, simply run:

p = parpool(c, 40);
% c: is the cluster profile object initialized using parcluster
% 40: because we want 40 workers

Important Note: You can only run one parpool job at a time. You need to make sure the parent job which launched the parpool job has a long enough wall time to accommodate the new job otherwise the parpool job will get terminated when the parent job ends.

To launch a batch job:

job1 = batch(c, @function, 1, {"arg1", "arg2"}, "Pool", 40); % launch batch job of 40 workers
% c: is the cluster profile object initialized using parcluster

wait(job1); % wait for job to finish

X = fetchOutputs(job1); %retrieve the output data from job

%job detail can be accessed by the job1 object including its status.

Here we launched a batch job to exectute @function. @function will be run on a parallel pool of 40 workers. 

Since batch does not block up your matlab program, you wan use the wait function to wait for your batch job(s) to finish before proceeding. The fetchOutputs function can be used to retrieve the outputs of the batch job.

 

The notable difference between parpool and batch is that you can run multiple batch jobs at a time and their duration is not tied to the parent job (the parent job can finish executing and the batch jobs will continue executing unlike parpool).

Please See the Running Concurrent Jobs section if running multiple jobs at the same time

please refer to the official MATLAB documentation for more details: parcluster  parpool  batch

 

Parfor

To parallelize a for-loop you can use a parfor-loop. 

A parfor-loop will run the different iterations of the loop in parallel by assigning the iterations to the workers in the pool. If multiple jobs are assigned to a worker then those jobs will be competed in serial by the worker. It is important to carefully assess and make good judgment calls on how many workers you want to request for the job.


To utilize a parfor-loop simply replace the for in a standard for-loop with parfor

%converting a standard for loop to a parfor looks as such:
for i=1:10
    %loop code
end

%replace the for with parfor
parfor i=1:10
    %loop code
end

 

Important note: parfor may complete the iterations out of order, so it is important that the iterations are not order dependent.

A parfor-loop is run synchronously, thus the MATLAB process is halted until all tasks for the workers are settled.

Important Limitations:

  • Cannot nest parfors inside of one another: This is because workers cannot start or access further parallel pools.
    • parfor-loops and for-loops can be nested inside one another (it is often a judgment call on whether it is better to nest a parfor inside a for-loop or vice versa).
%valid
for i=1:10
   parfor j=1:10
      %code
   end
end
%invalid: will throw error
parfor i=1:10
   parfor j=1:10
      %code
   end
end
  • Cannot have loop elements dependent on other iterations
    • Since, there is not guaranteed order of completion of iterations in a parfor-loop and workers cannot communicate with each other, each loop iteration must be independent.
  A = ones(1,100);
  parfor i = 1:100
       A(i) = A(i-1) + 1; %invalid iteration entry as the current iteration is dependent on the previous iteration
  end
  • step size must be 1
  parfor i = 0:0.1:1 %invalid because step side is not 1
       %code
  end

To learn more about par-for loops see the official matlab parfor documentation

 

Parfeval

Another way to run loops in parallel in MATLAB is to use parfeval loops. When using parfeval to run functions in the background it creates an object called a future for each function and adds the future object to a poll queue.

First, initialize a futures object vector with the number of expected futures. Preallocation of the futures vector is not required, but is highly recommended to increase efficiency: f(1:num_futures) = parallel.FevalFuture;

For each job, you can fill the futures vector with an instance of the future. Filling the vector allows you to get access to the futures later. f(index) = parfeval(@my_function, numOutputs, input1, input2);

  • @my_function is the pointer to the function I want to run
  • numOutputs is the integer represented number of returned outputs you need from my_function. Note: this does not need to match the actual number for outputs the function returns.
  • input1, input2, ... is the parameter list for my_function
%example code
f(1:10) = parallel.FevalFuture;
for i = 1:10
   f(i) = parfeval(@my_function, 1, 2);
end

when a future is created, it is added to a queue. Then the workers will takes futures from the queue to begin to evaluate them.

you can use the state property of a future to find out whether it is queuedrunning, or finishedf(1).State

you can manually cancel a future by running: cancel(f(1));

you can block off MATLAB until a certain future complete by using: wait(f(4));

when a future is finished you can check its error message is one was thrown by: f(1).Error.message

You can cancel all running and/or queued futures by (p is the parallel pool object):

cancel(p.FevalQueue.QueuedFutures);
cancel(p.FevalQueue.RunningFutures);

Processing worker outputs as they complete

One of the biggest strengths of parfeval  is its ability to run futures asynchronously (runs in the background without blocking the Matlab program). This allows you to fetch results from the futures as they get completed.

p = gcp; %luanch parallel pool with number of workers equal to availble cores

f(1:10) = parallel.FevalFuture; % initalize futures vector

for k = 1:10
    f(k) = parfeval(@rand, 1, 1000, 1); % lanch 10 futures which will run in background on parallel pool
end

results = cell(1,10); % create a results vector

for k = 1:10
    [completedK, value] = fetchNext(f); % fetch the next worker that finished and print its results
    results{completedK} = value;
    fprintf("got result with index: %d, largest element in vector is %f. \n", completedK, max(results{completedK}));
end

In this example above, as each  @rand  future gets completed by the workers, the fetchNext retrieves the returned data. 

MATLAB also provides functions such at afterEach and afterAll to process the outputs as workers complete futures.


Please refer to the official MATLAB documentation for more information on parfeval: parfeval and parfeval parallel pooling

Spmd

spmd stands for Single Program Multiple Data. The spmd block can be used to execute multiple blocks of data across multiple workers. Here is a simple example:

delete(gcp('nocreate')); %delete a parallel pool if one is already spun up
p = parpool(2); %create a pool of 2 workers

spmd
    fprintf("worker %d says hello world", spmdIndex); %have each worker print statement
end
%end of code
%output
Worker 1:
  worker 1 says hello world
Worker 2:
  worker 2 says hello world
%end of output

The spmdIndex variable can be used to access the index of each worker. spmd also allows for communication between workers via sending and receiving data. Additionally, data can be received by the MATLAB client from the workers. For more information on spmd and its functionality vist the Official MATLAB documentation

Submitting Single-Node Parallel MATLAB Jobs

When parallelizing on a single node, you can generate and run a parallel pool on the same node as the current job or interactive secession. 

Here is an example MATLAB script of submitting a parallel job to a single node:

p = parcluster('local');

% open parallel pool of 8 workers on the cluster node
parpool(p, 8);

spmd
   % assign each worker a print function
   fprintf("Worker %d says Hello", spmdIndex);
end

delete(gcp); % close the parallel pool
exit

Since we will only be using a single node, we will use the 'local' cluster profile. This will create a profile object p which will be the cluster profile of the job the command was run in. We also set the pool size be less than or equal to the number of cores on our compute node; In this case we will used 8. See cluster specifications to see the maximum number of cores on a single node for each cluster.

Now lets save this MATLAB script as "wokrer_hello.m" and write a Slurm batch script to submit and execute it as a job. "worker_hello.slurm" slurm script:

#!/bin/bash
#SBATCH --job-name=worker_hello         # job name
#SBATCH --cpus-per-task=8               # 8 cores
#SBATCH --output=worker_hello.log       # set output file
#SBATCH --time=00:10:00                 # 10 minutes wall time

# load Matlab module
module load matlab/r2023a

cd $SLURM_SUBMIT_DIR
#run matlab script
matlab -nodisplay -r worker_hello

In this script first we set a MATLAB module to the module path, in this example its MATLAB/r2023a. Then we make a call to execute the "worker_hello.m" MATLAB script. The -nodisplay flag is to prevent matlab from attempting to launch a GUI. In this script we requested 8 cores since our MATLAB script uses 8 workers. When performing single node parallelizations be mindful of the max number of cores each node has on the different clusters.

Then the job was submitted using sbatch -A <project-account> worker_hello.slurm through the command line. 

The output was then generated into the "worker_hello.log" file:

                          < M A T L A B (R) >
                 Copyright 1984-2023 The MathWorks, Inc.
            R2023a Update 2 (9.14.0.2254940) 64-bit (glnxa64)
                              April 17, 2023
                              
To get started, type doc.
For product information, visit www.mathworks.com.

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 8 workers.

Worker 1:
  Worker 1 says Hello
Worker 2:
  Worker 2 says Hello
Worker 3:
  Worker 3 says Hello
Worker 4:
  Worker 4 says Hello
Worker 5:
  Worker 5 says Hello
Worker 6:
  Worker 6 says Hello
Worker 7:
  Worker 7 says Hello
Worker 8:
  Worker 8 says Hello
  
Parallel pool using the 'Processes' profile is shutting down.

As we see a total of 8 workers were created and each printed their message in parallel.

Create Cluster Profile

Before we can parallelize matlab across multiple nodes we need to create a cluster profile. In the profile we can specify any arguments and adjust the settings of submitting jobs through MATLAB.

If you are running matlab r2019b and newer you can run configCluster to configure matlab with the profile of the cluster your job is running on:

configCluster % configer matlab with profile

c = parcluster; % get a handle to cluster profile


% set any additional properties

c.AdditionalProperties.WallTime = '00:10:00'; % set wall time to 10 mintues

c.AdditionalProperties.AccountName = 'PZS1234' % set account name


c.saveProfile % locally save the profile
When creating a profile you must set the AccountName and WallTime and make sure to save the profile. 

 

If the above method does not work, or you prefer to to use the GUI, then you can configure a cluster profile from the GUI. You must be running MATLAB r2023a and newer versions to be able to search for OSC's clusters.

1. First we need to launch a Matlab GUI through onDemand. See onDemand for more details.

2. Next within the MATLAB GUI, navigate to HOME->Environment->Parallel->Discover Clusters:

 

IMG_1.jpeg

3. Then check the "On your network" box. Then click Next.

IMG_2.png

4. If you started the Matlab GUI though onDemand then you should see the cluster of the session listed as such (I started mine through Pitzer so Pitzer is listed):

IMG_3.png

5. Now select the cluster and click Next. You should now have a screen like this:

IMG_4.png

6. Now check the "Set new cluster profile as default" box and then click Finish

7. Now if you click on HOME->Environment->Parallel->Select Parallel Environment you will be presented with a list of profiles available which you can toggle between. Your new profile that was just created should be listed. 

IMG_5.png

8. Now we need to edit the cluster profile to suit the needs of the job we want to submit. Go to HOME->Environment->Parallel->Create and Manage Clusters. Select the cluster profile you want to edit and then click edit. Most settings can be left as default but the following must be set: the AccountName and WallTime under the SCHEDULER PLUGIN must be set to your account name:

IMG_6.jpeg

If you want MATLAB to submit jobs with slurm parameters other than the default you may edit them in this menu.

When creating a profile you must set the AccountName and WallTime 

Validating Profile

If you run into any issues using your cluster profile you may want to validate your profile. Validating is not required, but may help debug any profile related issues.

To validate a profile:

  1. Within the MATLAB GUI, navigate to HOME->Environment->Parallel->Create and Manage Clusters:Screenshot 2023-08-11 at 12.07.04 PM.jpeg
  2. Select the profile you want to validate on the left side of the menu. Then select the Validation tab next to the Properties tab. Now in the Number of worker to use: box specifiy the number of cores you are using to run the OnDemand MATLAB GUI on. If you leave the box blank, then it will run the tests with more workers then cores available to your matlab session which will result in a failed validation.Screenshot 2023-08-11 at 12.11.01 PM.jpeg
  3. Next, click validate in the bottom right or top of the menu Screenshot 2023-08-11 at 12.23.05 PM.png

Make sure the AccountName and WallTime are both set in the cluster profile before validating! Make sure the number of worker used for validation is less than or equal to the number of cores available to the MATLAB session!

Submitting Multi-Node Parallel MATLAB Jobs

Before Submitting multi-node parallel jobs you must create a cluster profile. See Create Cluster Profile section above.

Now let's create a submit a multi-node parallel MATLAB job. Here is a matlab script:

configCluster % configer matlab with profile 

p = parcluster; % get a handle to cluster profile 

% set any additional properties 
p.AdditionalProperties.WallTime = '00:10:00'; % set wall time to 10 mintues 

p.AdditionalProperties.AccountName = 'PZS1234' % set account name 

p.saveProfile % locally save the profile

% if profile created using the "Discover Clusters" from the GUI then you can simply run: p = parcluster('Pitzer'); instead of the above code.


% open parallel pool of 80 workers
parpool(p, 80); % you must specify the number of workers you want

spmd
   fprintf("Worker %d says Hello", spmdIndex);
end

delete(gcp); % close the parallel pool
exit

In this example we opened a cluster profile called 'Pitzer'. This profile name should be the same as the cluster profile created above. We then launched another job using the parpool function with 80 workers onto the Pitzer cluster with the default settings (wall-time was set to 1 minutes instead of the default 1 hour). Since 80 workers is over the maximum number of cores per node, the Pitzer profile created using the steps above will automatically request 2 nodes for the job to accomodate the workers.

This script was saved in a file called "hello_multi_node.m".

Now a slurm script was created as follows:

#!/bin/bash
#SBATCH --job-name=hello_multi_node     # job name
#SBATCH --cpus-per-task=1               # 1 cores
#SBATCH --output=hello_multi_node.log   # set output file
#SBATCH --time=00:10:00                 # 10 minutes wall time

# load Matlab module
module load matlab/r2023a

cd $SLURM_SUBMIT_DIR
#run matlab script
matlab -nodisplay -r hello_multi_node

This job was allocated only 1 core. This is because the "hello_multi_node.m" will launch another job on the Pitzer cluster when calling parpool to exectute the parallel workers. Since the main entry matlab program does not need multiple nodes, we only allocated 1. 

Then the job was submitted using sbatch -A <project-account> hello_multi_node.slurm through the command line. 

The output was then generated into the "hello_multi_node.log" file:

                           < M A T L A B (R) >
                 Copyright 1984-2023 The MathWorks, Inc.
            R2023a Update 2 (9.14.0.2254940) 64-bit (glnxa64)
                              April 17, 2023

To get started, type doc.
For product information, visit www.mathworks.com.

Starting parallel pool (parpool) using the 'Pitzer' profile ...

additionalSubmitArgs =

   '--ntasks=80 --cpus-per-task=1 --ntasks-per-node=40 -N 2 --ntasks-per-core=1 -A PZS0711 --mem-per-cpu=4gb -t 00:01:00'

Connected to parallel pool with 80 workers.
Worker  1:
  Worker 1 says hello
Worker  2:
  Worker 2 says hello
Worker  3:
  Worker 3 says hello
Worker  4:
  Worker 4 says hello
Worker  5:
  Worker 5 says hello
.
.
.
Worker  80:
  Worker 80 says hello

Notice by the additionalSubmitArgs = line another job was launched with 2 nodes with 40 cores on each node. It is in this new job that the workers completed their tasks.

In this example we used parpool to launch a new parallel job, but batch can also be used. See MATLAB Parallel Functions and Tools for more information on the batch function

You can modify the properties of a cluster profile through code aswell through the c.AdditionalProperties attribute. This is helpful if you want to submit multiple batch jobs through a single Matlab program with different submit arguments.

c = parcluster('Pitzer'); % get cluster object

c.AdditionalProperties.WallTime = "00:15:00"; % sets the wall time to the c cluster object. Does not change the 'Pitzer' profile itself, only the local object.

c.saveProfile; % saves to the central 'Pitzer' profile.

Multithreading

Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

The system will run as many threads as there are cores on the nodes requested.

Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may include the option " -singleCompThread" when running MATLAB. An example is given below:

#!/bin/bash
#SBATCH --job-name disable_multithreading
#SBATCH --time=00:10:00
#SBATCH --nodes=1 --ntasks-per-node=40
#SBATCH --account=<project-account>

module load matlab
matlab -singleCompThread -nodisplay -nodesktop < hello.m
# end of example file

Using GPU in MATLAB

A GPU can be utilized for MATLAB. You can acquire a GPU for the job by

#SBATCH --gpus-per-node=1

for Pitzer. For more detail, please read here.

You can check the GPU assigned to you using:

gpuDeviceCount  # show how many GPUs you have
gpuDevice       # show the details of the GPU

To utilize a GPU, you will need to transfer the data from a standard CPU array to a GPU array. gpuArrays are a data structure which is stored on the GPU. Make sure the GPU has enough memory to hold this data. Even if the gpuArray fits in the GPU memory, make sure that any temporary arrays and data generated will also be able to fit on the GPU.

To create a GPU array:

X = [1,2,3]; %create a standard array
G = gpuArray(X); %transfer array over to gpu

To check if data is stored on the GPU run:

isgpuarray(G); %returns true or false

To transfer the GPU data back onto the host memory use:

Y = gather(G);


Note:

  • To reduce overhead time, limit the amount of data transfers between the host memory and GPU memory. For instance, many MATLAB functions allow you to create data directly on the GPU by specify the "gpuArray" parameter: gpu_matrix = rand(N, N, "gpuArray");
  • Gathering data from gpuArrays can be costly in terms of time and thus it is generally not necessary to gather the data unless you need to store it or the data needs processing to through non-gpu compatible functions.

When you have data in a GPU Array there are many built-in MATLAB functions which can run on the data. See list on the MATLAB website for a full list of compatible functions.

For more information about GPU programming for MATLAB, please read GPU Computing from Mathworks.

Running Concurrent Jobs

Concurrent jobs on OSC clusters

When you run multiple jobs concurrently, each job will try to access your preference files at the same time. It may create a race condition issue and may cause a negative impact on the system and the failure of your jobs. In order to avoid this issue, please add the following in your job script:

export MATLAB_PREFDIR=$TMPDIR

It will reset the preference directory to the local temporary directory, $TMPDIR. If you wish to start your Matlab job with the preference files you already have, add the following before you change MATLAB_PREFDIR.

cp -a ~/.matlab/{matlab version}/* $TMPDIR/

If you use matlab/r2020a, your matlab version is "R2020a".

Running MATLAB in Jupyter

MATLAB can be run on Jupyter Notebooks! First you will need to create a kernel, and then it will be available on Cardinal through a Jupyter session

Creating the Kernel

Run the following script, substituting {matlab version} with the version of MATLAB available on Cardinal you would like to run code for in Jupyter. You can have kernels of different versions as long as they are supported on Cardinal

~support/scripts/jupyter-matlab/create_jupyter_matlab_kernel {matlab version}

If you use matlab/r2024b your version is r2024b

You should receive a message like

## Creating the Jupyter MATLAB kernel in ~/.local/share/jupyter/kernels/jupyter_matlab_kernel_{matlab version};

A kernel directory will have been created at ~/.local/share/jupyter/kernels/jupyter_matlab_kernel_{matlab version}. To delete a kernel, you can remove that directory

rm -rf ~/.local/share/jupyter/kernels/jupyter_matlab_kernel_{matlab version}

Running Matlab in Jupyter

First, select the MATLAB kernel now available in Jupyter on OnDemand. It will be named "MATLAB Kernel {matlab version}"

MATLAB Kernel

Enter in any MATLAB code in the box. When you first run the code MATLAB will take time to start up, then you are ready to go!

Screenshot 2025-04-25 at 10.07.29 AM.png

 

References

Supercomputer: 
Service: 
Fields of Science: 

SPM

SPM is made freely available to the [neuro]imaging community, to promote collaboration and a common analysis scheme across laboratories. The software represents the implementation of the theoretical concepts of Statistical Parametric Mapping in a complete analysis package.

The SPM software is a suite of MATLAB (MathWorks) functions and subroutines with some externally compiled C routines. SPM was written to organise and interpret our functional neuroimaging data. The distributed version is the same as that we use ourselves.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

VERSION

Pitzer Cardinal
8

X

X
12.7771 X* X

* Current default version

spm/12.7771 comes with CONN 0.19 and xjview 9.7

spm/8 comes with CONN 0.19, xjview 9.7, and Marsbar 0.44

You can use module spider spm to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

SPM is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

SPM is free but copyright software, distributed under the terms of the GNU General Public Licence as published by the Free Software Foundation (either version 2, as given in file LICENCE.txt, or at your option, any later version). Further details on "copyleft" can be found at https://www.gnu.org/copyleft/. In particular, SPM is supplied as is. No formal support or maintenance is provided or implied.

Usage

Usage on Pitzer

Set-up

To configure your environment for use of SPM, run the following command: module load spm. The default version will be loaded. To select a particular AFNI version, use module load spm/version. For example, use module load spm/12.7771 to load SPM/12.7771.

SPM is a MATLAB suite, so you need to load MATLAB before you can use SPM:

module load matlab/r2020a
module load spm/12.7771
or
module load matlab/r2020a
module load spm/8
  

Note that spm/12.7771 comes with CONN 0.19 and xjview 9.7, and spm/8 comes with CONN 0.19, xjview 9.7, and Marsbar 0.44. Marsbar 0.44 doesn't support spm/12.7771.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

MRIQC

MRIQC is a program that provides automatic prediction of quality and visual reporting of MRI scans.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

Version Pitzer Ascend Cardinal
0.16.1   X X
23.1.0rc0   X X
24.1.0 X X X*
* Current default version

You can use module spider mriqc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

MRIQC is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

MRIQC uses the 3-clause BSD license; the full license is in the file LICENSE in the mriqc distribution. Open-source.

All trademarks referenced herein are property of their respective holders.

Copyright (c) 2015-2017, the mriqc developers and the CRN. All rights reserved.

Usage

Usage

Set-up

To configure your environment for use of mriqc, run the following command: module load mriqc/version. For example, use module load mriqc/0.16.1 to load MRIQC 0.16.1.

MRIQC is installed in a singularity container.  MRIQC_IMG environment variable contains the container image file path. So, an example usage would be

module load mriqc/0.16.1
singularity exec $MRIQC_IMG mriqc --version

For more information about singularity usages, please read OSC singularity page.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

MRIcroGL

MRIcroGL is medical image viewer that allows you to load overlays (e.g. statistical maps), draw regions of interest (e.g. create lesion maps).

Availability and Restrictions

Versions

MRIcroGL is available on Pitzer cluster. These are the versions currently available:

Version Pitzer Ascend Cardinal Notes
1.2.20220720 X X X*  

* Current default version

You can use module spider mricrogl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

mricrogl is available to all OSC users. Please review the license before you use. 

Publisher/Vendor/Repository and License Type

The Software has been developed for research purposes only and is not a clinical tool.

Copyright (c) 2014-2019 Chris Rorden. All rights reserved.

See more about the license for MRIcroGL at the GitHub repository here.

Usage

Usage on Pitzer

Set-up

To configure your environment for use of MRIcroGL, run the following command:  module load mricrogl/1.2.20220720. The default version will be loaded.

MRIcroGL is a GUI based software, so it requires an x11 connection. You can read about it from here for more details, but the simplest way to access the GUI is by using the OnDemand portal. Once you have an x11 connection, you can open the GUI by doing the following:

$ module load mricrogl/1.2.20220720
$ mricrogl.sif

MRIcroGL is installed in an apptainer container. For more information about apptainer usages, please read OSC apptainer page.

Further Reading

Supercomputer: 

MVAPICH

MVAPICH is a standard library for performing parallel processing using a distributed-memory model. 

Availability and Restrictions

Versions

The following versions of MVAPICH are available on OSC systems:

Version Cardinal Pitzer Ascend
3.0 X* X X
* Current default version

You can use module spider mvapich to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

MPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

NBCL, The Ohio State University/ Open source 

Usage

Set-up

To set up your environment for using the MPI libraries, you must load the appropriate module:

module load mvapich/3.0

You will get the default version for the compiler you have loaded.

Note:Be sure to swap the intel compiler module for the gnu module if you're using gnu.

Building With MPI

To build a program that uses MPI, you should use the compiler wrappers provided on the system. They accept the same options as the underlying compiler. The commands are shown in the following table.

Compiler Command
C mpicc
C++ mpicxx
FORTRAN 77 mpif77
Fortran 90 mpif90

For example, to build the code my_prog.c using the -O2 option, you would use:

mpicc -o my_prog -O2 my_prog.c

In rare cases you may be unable to use the wrappers. In that case you should use the environment variables set by the module.

Variable Use
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran 77 programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
$MPI_LIBS Use when linking your program to the MPI libraries.

For example, to build the code my_prog.c without using the wrappers you would use:

mpicc -c $MPI_CFLAGS my_prog.c

mpicc -o my_prog my_prog.o $MPI_LIBS

Batch Usage

Programs built with MPI can only be run in the batch environment at OSC. For information on starting MPI programs using the srun command, see Batch Processing at OSC.

Be sure to load the same compiler and mvapich modules at execution time as at build time.

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

MVAPICH2

MVAPICH2 is a standard library for performing parallel processing using a distributed-memory model. Also see MVAPICH.

Availability and Restrictions

Versions

The following versions of MVAPICH2 are available on OSC systems:

Version Ascend Cardinal
2.3.7-1 X X
* Current default version

You can use module spider mvapich2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

MPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

NBCL, The Ohio State University/ Open source 

Usage

Set-up

To set up your environment for using the MPI libraries, you must load the appropriate module:

module load mvapich2/2.3.7-1

You will get the default version for the compiler you have loaded.

Note:Be sure to swap the intel compiler module for the gnu module if you're using gnu.

Building With MPI

To build a program that uses MPI, you should use the compiler wrappers provided on the system. They accept the same options as the underlying compiler. The commands are shown in the following table.

C mpicc
C++ mpicxx
FORTRAN 77 mpif77
Fortran 90 mpif90

For example, to build the code my_prog.c using the -O2 option, you would use:

mpicc -o my_prog -O2 my_prog.c

In rare cases you may be unable to use the wrappers. In that case you should use the environment variables set by the module.

Variable Use
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran 77 programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
$MPI_LIBS Use when linking your program to the MPI libraries.

For example, to build the code my_prog.c without using the wrappers you would use:

mpicc -c $MPI_CFLAGS my_prog.c

mpicc -o my_prog my_prog.o $MPI_LIBS

Batch Usage

Programs built with MPI can only be run in the batch environment at OSC. For information on starting MPI programs using the srun or mpiexec command, see Batch Processing at OSC.

Be sure to load the same compiler and mvapich modules at execution time as at build time.

Known Issues

Large MPI job startup failure

Updated: Nov 2019
Versions Affected: Mvapich2/2.3 & 2.3.1
We have found that large MPI jobs may hang at startup with mvapich2/2.3 and mvapich/2.3.1 (on any compiler dependency) due to a known bug that has been fixed in release 2.3.2. If users experience this issue, please switch to mvapich2/2.3.2

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Mathematica

Mathematica is a mathematical computation program. It is capable in many areas of technical computing including but not limited to neural networks, machine learning, image processing, geometry, data science and visualizations.

Availability and Restrictions

Versions

Mathematica is available on the Pitzer Clusters. The versions currently available at OSC are:

  Pitzer
13.2.1 X

 

You can use module spider mathematica to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Use of Mathematica is open to academic Ohio State University usersOSC does not provide Mathematica licenses for outside of Ohio State University due to licensing restrictions. All users must be added to the system before using Mathematica. Please contact OSC Help to be granted access or for any license related questions.

Publisher/Vendor/Repository and License Type

Mathematica, commercial

Usage

Usage on Pitzer

Set-up on Pitzer

To load the default version of Mathematica module, use module load mathematica/13.2.1.

Running Mathematica

To run Mathematica, you should log into your osc account for OSC OnDemand. Then at the top of your screen navigate to the Interactive Apps dropdown menu. There you may select Mathematica and launch the task. After the application is available you can open and use Mathematica.

Alternatively, you may request an OSC OnDemand desktop and load Mathematica with the command module load mathematica/13.2.1. Then you can run Mathematica by typing the command mathematica

The command listed below will run Mathematica on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive Mathematica sessions on the login node for any significant computation. If your Mathematica script requires significant time, CPU power, or memory, you should run your code via the batch system.

Running Mathematica jobs with GPU

A GPU can be utilized for Mathematica. You can acquire a GPU for the job by

#SBATCH --gpus-per-node=1

for Pitzer. If running with an OnDemand desktop, select a GPU node to launch the desktop on.  For more detail, please read here.

 

For more information about GPU computing for Mathematica, please read GPU Computing from Wolfram.

Further Reading

 

Supercomputer: 
Service: 

Miniconda3

Miniconda3 is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.

Availability and Restrictions

Versions

Miniconda is available on the Ascend Cluster. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
24.1.2-py310 X X X*

* Current Default Version

You can use module spider miniconda3 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Miniconda3 is available to all OSC users, but all users are required to review and accept Anaconda, Inc. Terms of Service before accessing the software.

Publisher/Vendor/Repository and License Type

Anaconda Inc., Free use and redistribution under the terms of the EULA for Miniconda.

However, while miniconda3 itself is free use, it can be used to access proprietary channels and download software packages that have stricter license requirements.  See Anaconda, Inc. Terms of Service for details.

Usage

Supercomputer: 

NAMD

NAMD is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems. NAMD generally scales well on OSC platforms and offers a variety of modelling techniques. NAMD is file-compatible with AMBER, CHARMM, and X-PLOR.

Availability and Restrictions

Versions

The following versions of NAMD are available:

Version Pitzer Ascend Cardinal
3.0 X X X*
* Current default version
*  IMPORTANT NOTE: You need to load correct compiler and MPI modules before you use NAMD. In order to find out what modules you need, use module spider namd/{version} .

You can use  module spider namd  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

NAMD is available to all OSC users for academic purpose.

Publisher/Vendor/Repository and License Type

TCBG, University of Illinois/ Open source (academic)

Usage

Set-up

To load the NAMD software on the system, use the following command: module load namd/"version"  where "version" is the version of NAMD you require.

Using NAMD

NAMD is rarely executed interactively because preparation for simulations is typically performed with extraneous tools, such as, VMD.

Batch Usage

Sample batch scripts and input files are available here:

~srb/workshops/compchem/namd/

The simple batch script for Pitzer below demonstrates some important points. It requests 96 processors and 2 hours of walltime. If the job goes beyond 2 hours, the job would be terminated.

#!/bin/bash
#SBATCH --job-name apoa1
#SBATCH --nodes=2 --ntasks-per-node=48
#SBATCH --time=2:00:00
#SBATCH --account=<project-account>

module load intel/18.0.4
module load mvapich2/2.3.6
module load namd/3.0
# SLURM_SUBMIT_DIR refers to the directory from which the job was submitted.
# the following loop assumes you have the necessary .namd, .pdb, .psf, and .xplor files
# in the directory you are submitting the job from 
for FILE in *
do
    sbcast -p $FILE $TMPDIR/$FILE
done
# Use TMPDIR for best performance.
cd $TMPDIR
run_namd apoa1.namd
sgather -pr $TMPDIR $SLURM_SUBMIT_DIR/output
NOTE: ntaks-per-node should be a maximum of 48 on Pitzer.

Further Reading

Supercomputer: 
Service: 

NCCL

The NVIDIA Collective Communication Library (NCCL) implements multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and Networking. NCCL provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter as well as point-to-point send and receive that are optimized to achieve high bandwidth and low latency over PCIe and NVLink high-speed interconnects within a node and over NVIDIA Mellanox Network across nodes.

Availability and Restrictions

Versions

NCLL is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2.19.3-1 X X X*

* Current default version

You can use module spider nccl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

NCCL is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

NVIDIA, see NVIDIA's links listed here for licensing.

SLA
This document is the Software License Agreement (SLA) for NVIDIA NCCL. The following contains specific license terms and conditions for NVIDIA NCCL. By accepting this agreement, you agree to comply with all the terms and conditions applicable to the specific product(s) included herein.
 
BSD License
This document is the Berkeley Software Distribution (BSD) license for NVIDIA NCCL. The following contains specific license terms and conditions for NVIDIA NCCL open sourced. By accepting this agreement, you agree to comply with all the terms and conditions applicable to the specific product(s) included herein.

Usage

Performance

The performance results were obtained by running NVIDIA NCCL Tests. The tests were built with NCCL 2.19.3, CUDA 12, and OpenMPI 5. Each performance value represents the average of five runs using a 512MB message size. The total number of ranks for each test was configured as follows:

  • Single-node Allreduce: -g $SLURM_GPUS_PER_NODE -t 1
  • Single-node SendRecv: -g 2 -t 1
  • Node-to-node: srun -N 2 --ntasks-per-node=1 with -g 1 -t 1

Note: For Ascend dual-GPU nodes, the environment variable NCCL_P2P_DISABLE was set to 1 due to a known issue.

Cluster Single Node Node to Node
  SendRecv Allreduce SendRecv Allreduce
Cardinal 124 GB/s 240 GB/s 28.8 GB/s 46.7 GB/s
Ascend (quad) 72 GB/s 144 GB/s 6.3 GB/s 6.3 GB/s
Ascend (dual) 11.8 GB/s 12.0 GB/s 9.5 GB/s 9.5 GB/s
Pitzer 8.5 GB/s 7.3 GB/s 5.3 GB/s 8.8 GB/s

Known Issues

Tag: 
Supercomputer: 
Service: 
Technologies: 

NVHPC

NVHPC, or NVIDIA HPC SDK, C, C++, and Fortran compilers support GPU acceleration of HPC modeling and simulation applications with standard C++ and Fortran, OpenACC® directives, and CUDA®. GPU-accelerated math libraries maximize performance on common HPC algorithms, and optimized communications libraries enable standards-based multi-GPU and scalable systems programming. Performance profiling and debugging tools simplify porting and optimization of HPC applications, and containerization tools enable easy deployment on-premises or in the cloud. With support for NVIDIA GPUs and Arm, OpenPOWER, or x86-64 CPUs running Linux, the HPC SDK provides the tools you need to build NVIDIA GPU-accelerated HPC applications.

Availability and Restrictions

Versions

The versions currently available at OSC are:

Versions Pitzer Ascend Cardinal
24.11 X X X
25.1 X X X*

* Current Default Version

You can use module spider nvhpc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

NVHPC is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

NVIDIA, Please review the license agreement carefully before use.

Usage

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

NWChem

NWChem aims to provide its users with computational chemistry tools that are scalable both in their ability to treat large scientific computational chemistry problems efficiently, and in their use of available parallel computing resources from high-performance parallel supercomputers to conventional workstation clusters.

Availability and Restrictions

Versions

NWChem is not currently available on any clusters at the OSC.

Version Pitzer Cardinal
7.2.3 X X

You can use module spider nwchem to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

NWChem is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

EMSL, Pacific Northwest National Lab., Open source

Usage

Set-up

To configure your environment for use of NWChem, you must first load the gcc and mvapich libraries, then use module load nwchem/version. To see which modules must be loaded for a specific version, use module spider nwchem/version. For example: to load NWChem 7.2.3, run the following command: module load intel/2021.10.0 mvapich/4.1 nwchem/7.2.3.

Performance

The performance results were obtained by running the C240 benchmark using NWChem version 7.2.3.

Cluster # CPUs Build Dependencies CPU Time
Cardinal 96 intel/2021.10.0 mvapich/4.1 1948s
Pitzer 48 intel/2021.10.0 mvapich/4.1 2730s

Further Reading

Supercomputer: 
Service: 

Ncview

Ncview is a visual browser for netCDF format files. Typically you would use ncview to get a quick and easy, push-button look at your netCDF files. You can view simple movies of the data, view along various dimensions, take a look at the actual data values, change color maps, invert the data, etc.

Availability and Restrictions

Versions

The following versions of Ncview are available on OSC clusters:

Version Pitzer Ascend Cardinal
2.1.10 X X X*
* Current default version

You can use  module spider ncview to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Ncview is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

David W. Pierce, Open source

Usage

Usage

Set-up

To configure your environment for use of Ncview, run the following command: module load ncview/version. For example, use module load ncview/2.1.7 to load Ncview 2.1.7.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

NetCDF

NetCDF (Network Common Data Form) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netcdf library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data.

Availability and Restrictions

Versions

NetCDF is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
1.12.3 P P P
4.3.1 S S S
4.6.1 F F F
4.8.1 C C C
* Current default version
P = parallel-netcdf, S = netcdf-cxx4, F = netcdf-fortran, C = netcdf-c

 

You can use module spider netcdf to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

The netcdf module is split into three parts: netcdf-c for the C interface, netcdf-fortran for the Fortran interface and netcdf-cxx4 for the C++ interface. You can load netcdf-c with  module load netcdf-c, netcdf-cxx4 with module load netcdf-cxx4, and netcdf-fortran with module load netcdf-fortran

Access

NetCDF is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

University Corporation for Atmospheric Research, Open source

Usage

Usage on Cardinal

Set-up

Initalizing the system for use of the NetCDF is dependent on the system you are using and the compiler you are using. To load the default NetCDF C interface, run the following command: module load netcdf-c. The pnetcdf module is parallel-netcdf on Cardinal. To load parallel-netcdf, run the following command: module load parallel-netcdf 

Building With NetCDF

With any of the netcdf libraries loaded, the following environment variables will be available for use:

VARIABLE USE
$NETCDF_CFLAGS Use during your compilation step for C or C++ programs.
$NETCDF_FFLAGS Use during your compilation step for Fortran programs.
$NETCDF_LIBS Use when linking your program to NetCDF.

 

Similarly, when the parallel-netcdf module is loaded, the following environment variables will be available:

VARIABLE USE
$PNETCDF_CFLAGS Use during your compilation step for C programs.
$PNETCDF_FFLAGS Use during your compilation step for Fortran programs.
$PNETCDF_LIBS Use when linking your program to NetCDF.
 

 

For example, to build the code myprog.c with the netcdf library you would use:

icc -c $NETCDF_CFLAGS myprog.c
icc -o myprog myprog.o $NETCDF_LIBS

Batch Usage

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Non-interactive Batch Job (Serial Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. You must load the netcdf or parallel-netcdf module in your batch script before executing a program which is built with the netcdf library. Below is the example batch script that executes a program built with NetCDF C interface:
#!/bin/bash 
#SBATCH --job-name=job-name
#SBATCH --nodes=1 --ntasks-per-node=48 
#SBATCH --account <project-account> 

module load netcdf-c 
cp foo.dat $TMPDIR 
cd $TMPDIR 
appname < foo.dat > foo.out 
cp foo.out $SLURM_SUBMIT_DIR

Further Reading

See Also

Tag: 
Supercomputer: 
Service: 

Neuropointillist

Neuropointillist is an in-development R package which defines functions to help scientists to run voxel-wise models using R neuroimaging data.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

Version Pitzer Ascend Cardinal
0.0.0.9000 X X X*
* Current default version

You can use module spider neuropointillist to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Neuropointillist is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Free and open source.

MIT License

Copyright (c) 2018 Tara Madhyastha

Full license information available through LICENSE file in the software.

Usage

Set-up

To configure your environment for use of Neuropointillist, run the following command: module load neuropointillist/version. For example, use  module load neuropointillist/0.0.0.9000 to load Neuropointillist 0.0.0.9000.

Neuropointillist is an R package, so you need to load the R module before you can use it in R.

module load R/4.0.2-gnu9.1
module load neuropointillist/0.0.0.9000

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Nextflow

Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. Nextflow is based on the dataflow programming model which simplifies complex distributed pipelines.

Availability and Restrictions

Versions

Nextflow is available on the Pitzer clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
24.10.4 X X X
25.04.6 X X X

You can use module spider nextflow to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Nextflow is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Developed by Seqera and distributed under Apache 2.0 license, open-source

Usage

Set-up

To load the default Nextflow library, run the following command:  module load nextflow/version. For example, use module load nextflow/21.10.3 to load Nextflow version 21.10.3. You can use module spider nextflow to view available modules.

Batch Usage

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Further Reading

Tag: 
Supercomputer: 
Service: 

Nodejs

Nodejs is used to create server-side web applications, and it is perfect for data-intensive applications since it uses an asynchronous, event-driven model

Availability and Restrictions

Versions

Nodejs is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
20.12.0 X X X*
22.12.0 X X X
* Current Default Version

You can use module spider node-js to view available modules. Feel free to contact OSC Help if you need other versions for your work.

Access

Nodejs is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

OpenJS Foundation, Open source 

Usage

Usage on Pitzer

Set-up

To load the default Nodejs library, run the following command: module load node-js/version. For example, use module load node-js/14.17.3 to load Nodejs version 14.17.3. You can use module spider node-js to view available modules.

Nodejs version 18.18.2 Usage

Nodejs verion 18.18.2 is contianerized. To learn more about containers see: HOWTO: Use Docker and Apptainer/Singularity Containers at OSC

To use nodejs/18.18.2 simply run:

node

or 

apptainer exec $NODE_IMG node 

Both of the above commands will also work with additonal command line arguments such as node script.js and apptainer exec $NODE_IMG node script.js.

 

If you need to use npm with node/18.18.2 then you will need to first open a shell in the container. To do so run:

node_shell

or 

apptainer shell $NODE_IMG

Now within this shell you can run node and npm.

Batch Usage

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Further Reading

Tag: 
Supercomputer: 
Service: 

ORCA

ORCA is an ab initio quantum chemistry program package that contains modern electronic structure methods including density functional theory, many-body perturbation, coupled cluster, multireference methods, and semi-empirical quantum chemistry methods. Its main field of application is larger molecules, transition metal complexes, and their spectroscopic properties. ORCA is developed in the research group of Frank Neese. Visit ORCA Forum for additional information.

We have found that several ORCA 5 jobs requiring heavy I/O load on scratch/project filesystems are causing performance issues and affecting the performance of the filesystems. For optimal performance, we recommend to run such ORCA jobs on a local disk ($TMPDIR), as discussed in the ORCA forum:

    https://orcaforum.kofo.mpg.de/viewtopic.php?f=8&t=10935&p=45270&hilit=di...
    https://orcaforum.kofo.mpg.de/viewtopic.php?f=9&t=10835&p=44967&hilit=di...

We also recommend using ORCA 4.2.1 unless ORCA 5 is necessary for your job. To run an ORCA job using $TMPDIR, please refer to the example in the Usage section below.
To avoid potential memory issues, it is important to tune the %maxcore value based on the number of cores you request. Plese refer to the "Best practices" section in the Usage guidelines below for more details.

Availability and Restrictions

Versions

ORCA is available on the OSC clusters. These are the versions currently available:

Version Pitzer Ascend Cardinal Notes
5.0.4 X X X openmpi/5.0.2
6.0.1 X   X openmpi/5.0.2
6.1.0     X* openmpi/5.0.2
* Current default version. The notes indicate the MPI module likely to produce the best performance, but see the Known Issue below named "Bind to CORE".

You can use module spider orca to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

ORCA is available to OSC academic users; users need to sign up ORCA Forum. You will receive a registration confirmation email from the ORCA management. Please contact OSC Help with the confirmation email for access.

Publisher/Vendor/Repository and License Type

ORCA, Academic (Computer Center)

Usage

Usage

Set-up

ORCA usage is controlled via modules. Load one of the ORCA modulefiles at the command line, in your shell initialization script, or in your batch scripts. To load the a particular software version, use module load orca/{version}. For example, use module load orca/4.2.1to load ORCA version 4.2.1.

IMPORTANT NOTE: You need to load correct compiler and MPI modules before you use ORCA. In order to find out what modules you need, use module spider orca/{version}.

Batch Usage

When you log into pitzer.osc.edu or pitzer.osc.edu, you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

For an interactive batch session one can run the following command:

sinteractive -A <project-account> -n 1 -t 00:20:00

which requests one core (-n 1), for a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

Non-interactive Batch Job

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script for a parallel run:

#!/bin/bash
#SBATCH --job-name=orca_mpi_test
#SBATCH --time=0:10:0
#SBATCH --nodes=2 --ntasks-per-node=<number-of-cores-per-node>
#SBATCH --account=<project-account>

module reset
module load openmpi/3.1.6-hpcx
module load orca/4.2.1
module list

sbcast -p h2o_b3lyp_mpi.inp $TMPDIR/h2o_b3lyp_mpi.inp
cd $TMPDIR
$ORCA/orca h2o_b3lyp_mpi.inp > $SLURM_SUBMIT_DIR/h2o_b3lyp_mpi.out

Please note that the <number-of-cores-per-node> cannot exceed the maximum cores per node. You can refer to Cluster Computing for the maximum number for each cluster.

Best practices

Set correct value for %maxcore

In general, it is recommended to utilize 3000, which is 75% of the usable memory per core on each cluster. However, you may need to increase %maxcore due to the methods and the modular system. In this case, you can decrease the number of cores for the same job. For example, if you have the following script to run an 80-core ORCA job on two Pitzer 40-core nodes:

#!/bin/bash
#SBATCH --nodes=2 --ntasks-per-node=40

module reset
module load openmpi/3.1.6-hpcx
module load orca/4.2.1
module list

sbcast -p h2o_b3lyp_mpi.inp $TMPDIR/h2o_b3lyp_mpi.inp
cd $TMPDIR
$ORCA/orca h2o_b3lyp_mpi.inp > $SLURM_SUBMIT_DIR/h2o_b3lyp_mpi.out

If you need to increase %maxcore to 4000, you can run ORCA with 60 cores (30 cores per node) in the same job script by replacing the ORCA command line with:

$ORCA/orca h2o_b3lyp_mpi.inp "--npernode=30" > $SLURM_SUBMIT_DIR/h2o_b3lyp_mpi.out

Further Reading

Scratch Storage information is availiable from the Storage Documentation

 

Supercomputer: 
Service: 

Ollama

Ollama is an open-source inference server for large language models (LLMs).  This module also includes Open-WebUI, which provides an easy-to-use web interface.

Ollama is in early user testing phase - not all functionality is guaranteed to work.  Contact oschelp@osc.edu with any questions.
Ollama is not currently suitable for use with protected or sensitive data - do not use if you need protected data service. See https://www.osc.edu/resources/protected_data_service for more details.

Availability and Restrictions

Versions

Ollama is available on OSC Clusters. The versions currently available at OSC are:

Version Cardinal Ascend
0.5.13 X X
0.11.3 X X
0.12.5 X X
0.13.1 X X

 

You can use module spider ollama to view available modules for a given machine.

Access:

All OSC users may use Ollama and Open-WebUI, but individual models may have their own license restrictions.

Publisher/Vendor/Repository and License Type

https://github.com/ollama/ollama, MIT license.

https://github.com/open-webui/open-webui, BSD-3-Clause license.

Prerequisites

  • GPU Usage: Ollama should be run with a GPU for best performance. 
  • OnDemand Desktop Session: If using the Open-WebUI web interface, you will need to first start an OnDemand Desktop session on Cardinal/Ascend with GPU.

Due to the need for GPUs, we recommend not running Ollama on login nodes nor OnDemand lightweight desktops.

Running Ollama and Open-WebUI Overview

1. Load module

2. Start Ollama

3. Start Open-WebUI

 

Commands

Ollama is available through the module system and must be loaded prior to running any of the commands below:

loading ollama module:
module load ollama/0.13.1
Starting ollama:
ollama_start

This will print out a port number for the Ollama service. E.g.,

Ollama port: 61234

Starting open-webui:
open_webui_start

This will print out a port number for the Open_WebUI service. E.g.,

Open_WebUI port: 51234

Port numbers are only examples - your port numbers will differ from the ones above.

Ollama must be running for Open-WebUI to connect.  Starting Open-WebUI will automatically open a browser.

Take note of your port numbers, as you will need them if you close your browser.
Stopping Ollama and Open-WebUI:

Ollama and Open-WebUI can be manually stopped with the following commands:

ollama_stop
open_webui_stop

They are also killed upon module unload.  If you want to stop the services, you can simply unload the ollama module:

module unload ollama/0.13.1

Model Management

By default, Ollama uses a central, read-only model repository defined by OLLAMA_MODELS

However, you can use custom models and manage your own set of models by setting OLLAMA_MODELS to an existing path you have write access to, such as a project directory or scratch space.  This must be done prior to starting Ollama.

export OLLAMA_MODELS=/fs/project/ABC1234/ollama/models
ollama_start
installing a model:
ollama_pull <modelname>

The list of supported models can be found at ollama.com/library. Ollama must be running prior to pulling a new model. 

Downloading large LLMs can exceed your disk space quota.  Check model sizes before downloading!


Some models require licensing agreements or are otherwise restricted and require a Hugging Face account and login.  With the Ollama module loaded, use the huggingface-cli tool to login:

hf auth login

For more details, see https://huggingface.co/docs/huggingface_hub/en/guides/cli.

 

Deleting a model:
ollama_rm <modelname>

Ollama must be running prior to deleting model.  You can only delete models if you are using a custom OLLAMA_MODELS path that you have write access to.

 

Interactive vs. Batch Usage

Ollama can be used interactively by loading the module and starting the service(s) as described above.

Requesting a GPU-enabled desktop session and using Open-WebUI is one possible use case.

The Ollama module can also be used in batch mode by loading the module in your batch script.  For example, you may want to run offline inference by running a script that relies on an inference endpoint.

Ollama provides an OpenAI API-compliant API endpoint, and can be accessed by Open-WebUI or another OpenAI API-compliant client, meaning you can bring your own clients or write your own.  As long as you can send requests to localhost:OLLAMA_PORT, this should work and support a wide variety of workflows. 

For the most up-to-date API compatibility information (and more examples), see: Ollama API docs and Open-WebUI API docs.  OpenAI API chat completion docs are useful as a reference, but Ollama does not currently support the complete OpenAI API, including tools and responses.

Here is a basic Python example using the OpenAI package:

import os
from openai import OpenAI

ollama_port = os.getenv("OLLAMA_PORT")

client = OpenAI( base_url = f"http://localhost:{ollama_port}/v1", api_key="") 

response = client.chat.completions.create(
    model = "gemma3:12b",
    messages = [
        {"role": "developer", "content": "talk like a pirate"},
        {"role": "user", "content": "how do I check a Python object's type?"}
     ]
)

For more advanced API usage example with asynchronous requests, see this GitHub project: OSC/async_llm_api 

Please note this software is in early user testing and might not function as desired.  Please reach out to oschelp@osc.edu with any issues.

Jupyter Usage

This is under development - contact oschelp@osu.edu if you're interested in this functionality.

 

Supercomputer: 
Technologies: 

OpenACC

OpenACC is a standard for parallel programming on accelerators, such as Nvidia GPUs and Intel Phi. It consists primarily of a set of compiler directives for executing code on the accelerator, in C and Fortran. OpenACC is currently only supported by the PGI compilers installed on OSC systems.

Availability and Restrictions

OpenACC is available to all OSC users. It is supported by the PGI compilers. If you have any questions, please contact OSC Help.

Usage

Set-up

OpenACC support is built into the compilers. There is no separate module to load.

Building With OpenACC

To build a program with OpenACC, use the compiler flag appropriate to your compiler. The correct libraries are included implicitly.

Compiler Family Flag
PGI -acc -ta=nvidia -Minfo=accel

Batch Usage

An OpenACC program will not run without an acelerator present. You need to ensure that your PBS resource request includes GPUs. For example, to run an OpenACC program on Owens, your resource request should look something like this: #PBS -l nodes=1:ppn=28:gpus=2.

Further Reading

See Also

Service: 
Fields of Science: 

OpenCV

OpenCV is an open-source library that includes several hundreds of computer vision algorithms.

Availability and Restrictions

Versions

Version Ascend
3.4.6 X#
* Current default version; # System version

You can use module spider opencv to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

OpenCV is available to all OSC users.

Publisher/Vendor/Repository and License Type

OpenCV versions after 4.5.0 fall under the Apache 2 license. Full details are available here.

Usage

Legacy usage

Set-up

The legacy system version does not need to be loaded.   Keep in mind that it dates to many years ago.  In general, it should be used with other tools from the same era, e.g., the system compiler version, which can be selected for your environment via module load gnu/4.8.5 on Pitzer.

Usage on Pitzer

Set-up on Pitzer

To load the default version of the OpenCV module which initalizes your environment for non legacy OpenCV, use module load opencv. To select a particular OpenCV version, use module load opencv/version. For example, use module load opencv/4.5.4 to load OpenCV 4.5.4.

In general users should employ the helper variables defined by an OpenCV module, e.g., module load gnu/9.1.0 cuda/11.2.2 opencv/4.5.4; g++ $OPENCV_INCLUDE $OPENCV_LIB bla bla.  A complete  example is available; for its location and other installation details see the output of module spider opencv/4.5.4.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

OpenFOAM

OpenFOAM is a suite of computational fluid dynamics applications. It contains myriad solvers, both compressible and incompressible, as well as many utilities and libraries.

Availability and Restrictions

Versions

The following versions of OpenFOAM are available on OSC clusters:

Version Pitzer Ascend Cardinal
2312 X X X
2412     X

The location of OpenFOAM may be dependent on the compiler/MPI software stack, in that case, you should use one or both of the following commands (adjusting the version number) to learn how to load the appropriate modules:

module spider openfoam
module spider openfoam/2306

Feel free to contact OSC Help if you need other versions for your work.

Access 

OpenFOAM is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

OpenFOAM Foundation, Open source

Basic Structure for an OpenFOAM Case

The basic directory structure for an OpenFOAM case is:

/home/yourusername/OpenFOAM_case
|-- 0
|-- U
|-- epsilon
|-- k
|-- p
`-- nut 
|-- constant
|-- RASProperties
|-- polyMesh
|   |-- blockMeshDict
|   `-- boundary
|-- transportProperties
`-- turbulenceProperties
|-- system
|-- controlDict
|-- fvSchemes
|-- fvSolution
`-- snappyHexMeshDict

IMPORTANT: To run in parallel, you need to also create the decomposeParDict file in the system directory. If you do not create this file, the decomposePar command will fail.

Usage

Usage on Pitzer

Setup on Pitzer

To configure the Pitzer cluster for the use of OpenFOAM 5.0, use the following commands:
module load openmpi/3.1.0-hpcx # currently only 5.0 is installed using OpenMPI libraries
module load openfoam/5.0

Batch Usage on Pitzer

Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems.

On Pitzer, refer to Queues and Reservations for Pitzer and Scheduling Policies and Limits for more info. 

Interactive Batch Session

For an interactive batch session on Pitzer, one can run the following command:

sinteractive -A <project-account> -N 1 -n 40 -t 1:00:00

which gives you 1 node (-N 1), 40 cores (-n 40) with 1 hour (-t 1:00:00). You may adjust the numbers per your need. 

Non-interactive Batch Job (Serial Run)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (job.txt) for a serial run:

#!/bin/bash
#SBATCH --job-name serial_OpenFOAM 
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH --time 24:00:00 
#SBATCH --account <project-account>

# Initialize OpenFOAM on Pitzer Cluster
module load openmpi/3.1.0-hpcx
module load openfoam

# Copy files to $TMPDIR and move there to execute the program
cp * $TMPDIR
cd $TMPDIR
# Mesh the geometry
blockMesh
# Run the solver
icoFoam
# Finally, copy files back to your home directory
cp * $SLURM_SUBMIT_DIR

To run it via the batch system, submit the job.txt file with the following command:

sbatch job.txt
Non-interactive Batch Job (Parallel Run)

Below is the example batch script (job.txt) for a parallel run:

#!/bin/bash
#SBATCH --job-name parallel_OpenFOAM
#SBATCH --nodes=2 --ntasks-per-node=40
#SBATCH --time=6:00:00
#SBATCH --account <project-account>

# Initialize OpenFOAM on Ruby Cluster
# This only works if you are using default modules
module load openmpi/3.1.0-hpcx 
module load openfoam/5.0

# Mesh the geometry
blockMesh
# Decompose the mesh for parallel run
decomposePar
# Run the solver
mpiexec simpleFoam -parallel 
# Reconstruct the parallel results
reconstructPar

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

OpenMP

OpenMP is a standard for parallel programming on shared-memory systems, including multicore systems. It consists primarily of a set of compiler directives for sharing work among multiple threads. OpenMP is supported by all the Fortran, C, and C++ compilers installed on OSC systems.

Availability and Restrictions

OpenMP is available to all OSC users. It is supported by the Intel, PGI, and gnu compilers. If you have any questions, please contact OSC Help.

Usage

Set-up

OpenMP support is built into the compilers. There is no separate module to load.

Building With OpenMP

To build a program with OpenMP, use the compiler flag appropriate to your compiler. The correct libraries are included implicitly.

Compiler Family Flag
Intel -qopenmp or  -openmp
gnu -fopenmp
PGI -mp

Batch Usage

An OpenMP program by default will use a number of threads equal to the number of processor cores available. To use a different number of threads, set the environment variable OMP_NUM_THREADS.

Further Reading

See Also

Service: 
Fields of Science: 

OpenMPI

MPI is a standard library for performing parallel processing using a distributed memory model. The Pitzer, Ascend, and Cardinal clusters at OSC can use the OpenMPI implementation of the Message Passing Interface (MPI).

Availability and Restrictions

Versions

Installations are available for the Intel, PGI, and GNU compilers. The following versions of OpenMPI are available on OSC systems:

Version Pitzer Ascend Cardinal Notes
4.1.6     X  
5.0.2 X X X*  
* Current default version

You can use module spider openmpi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

OpenMPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

https://www.open-mpi.org, Open source

Usage

Setup on OSC Clusters

To set up your environment for using the MPI libraries, you must load the appropriate module. On any OSC system, this is performed by: module load openmpi/version For example, use module load openmpi/5.0.2 to load openMPI 5.0.2.

Building With MPI

To build a program that uses MPI, you should use the compiler wrappers provided on the system. They accept the same options as the underlying compiler. The commands are shown in the following table:

C mpicc
C++ mpicxx
FORTRAN 77 mpif77
Fortran 90 mpif90

For example, to build the code my_prog.c using the -O2 option, you would use:

mpicc -o my_prog -O2 my_prog.c

In rare cases, you may be unable to use the wrappers. In that case, you should use the environment variables set by the module.

Variable Use
$MPI_CFLAGS Use during your compilation step for C programs.
$MPI_CXXFLAGS Use during your compilation step for C++ programs.
$MPI_FFLAGS Use during your compilation step for Fortran 77 programs.
$MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.

Batch Usage

Programs built with MPI can only run in the batch environment at OSC. For information on starting MPI programs using the command srun see Job Scripts.

Be sure to load the same compiler and OpenMPI modules at execution time as at build time.

Run a MPI program 

SRUN

We recommend the command srun as the default MPI launcher. Please refer to Pitzer Programming Environment for detail.

Further Reading

See Also

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

ParaView

ParaView is an open-source, multi-platform application designed to visualize data sets of size varying from small to very large. ParaView was developed to support distributed computational models for processing large data sets and to create an open, flexible user interface.

Availability and Restrictions

Versions

ParaView is available on the Cardinal clusters. The versions currently available at OSC are:

Version Cardinal
5.13.0 X*
* Current default version

You can use module spider paraview to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

ParaView is available for use by all OSC users.

Publisher/Vendor/Repository and License Type

https://www.paraview.org, Open source

Usage

Usage on Cardinal

Set-up

To load the default version of ParaView module, use module load paraview.  Following a successful loading of the ParaView module, you can access the ParaView program:
paraview

Using ParaView with OSC OnDemand

Using ParaView with OSC OnDemand requires VirtualGL. To begin, connect to OSC OnDemand and luanch a virtual desktop, either a Virtual Desktop Interface (VDI) or an Interactive HPC desktop. In the desktop open a terminal and load the ParaView and VirtualGL modules with module load paraview and module load virtualgl. You can then access the ParaView program with:

vglrun paraview

Note that using ParaView with OSC OnDemand does not work on all clusters.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Parallel-netCDF

Parallel-netCDF is a library providing high-performance parallel I/O while still maintaining file-format compatibility with  Unidata's NetCDF, specifically the formats of CDF-1 and CDF-2. Although NetCDF supports parallel I/O starting from version 4, the files must be in HDF5 format. PnetCDF is currently the only choice for carrying out parallel I/O on files that are in classic formats (CDF-1 and 2). In addition, PnetCDF supports the CDF-5 file format, an extension of CDF-2, that supports more data types and allows users to define large dimensions, attributes, and variables (>2B elements).

Availability and Restrictions

Versions

The following versions of PnetCDF are available at OSC:

Version Pitzer Ascend Cardinal
1.12.3 X X X*
* Current default version

You can use module spider parallel-netcdf to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work

Access

Parallel-netCDF is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Northwestern University and Argonne National Lab., Open source

Further Reading

Supercomputer: 
Service: 

Perl

Perl is a family of programming languages.

Availability and Restrictions

Versions

Perl is available on all OSC clusters. Only one version is available at any given time. To find out the current version, run:

perl --version

Feel free to contact OSC Help if you need other versions for your work.

Access

Perl is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

https://www.perl.org, Open source

Usage

Each cluster has a version of Perl that is part of the Operating System (OS). Some perl scripts (usually such files have a .pl extension) may require particular Perl Modules (PMs) (usually such files have a .pm extension). In some cases particular PMs are not part of the OS; in those cases, users should install those PMs; for background and a general recipe see HOWTO: Install your own Perl modules. In other cases a PM may be part of the OS but in an unknown location; in that case an error like this is emitted: Can't locate Shell.pm in @INC; and users can rectify this by locating the PM with the command locate Shell.pm and then adding that path to the environment variable PERL5LIB, e.g. in csh syntax: setenv PERL5LIB "/usr/share/perl5/CPAN:$PERL5LIB"

Further Reading

Supercomputer: 

Picard

Picard is a set of command line tools for manipulating high-throughput sequencing (HTS) data and formats such as SAM/BAM/CRAM and VCF.

Availability and Restrictions

Versions

The following versions of Picard are available on OSC clusters:

Version Pitzer Ascend Cardinal
3.0.0 X X X*
* Current default version

You can use  module spider picard to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Picard is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

The Broad Institute, Open source

Usage

Usage on Pitzer

Set-up

To configure your environment for use of Picard, run the following command: module load picard/version. For example, use module load picard/3.0.0 to load version 3.0.0.

Usage

This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable.

From module load picard/3.0.0, a new environment variable, PICARD, will be set. Thus, users can use the software by running the following command:  java -jar $PICARD {other options}.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Podman

Podman is an Open Containers Initiative (OCI)‑compliant, daemonless and rootless container tool developed by Red Hat. Unlike Docker, Podman operates without a central daemon and supports secure rootless execution, making it well‑suited for HPC environments and schedulers such as Slurm. 

Availability and Restrictions

Versions

Podman is available on all OSC clusters. Only one version is available at any given time. To find out the current version:

podman -v

Access

Podman is available to all OSC users.

Publisher/Vendor/Repository and License Type

Red Hat/Apache License 2.0

Usage

Set-up

No setup is required. You can use Podman directly on all clusters.

Using Podman

In addition to the podman command, Podman provides a script named docker that emulates the Docker CLI by executing Podman commands. It also creates symbolic links from all Docker CLI man pages to their corresponding Podman documentation. For example,

[pitzer-login01]$ docker run busybox echo "Hello from Busybox!"...
Resolved "busybox" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 90b9666d4aed done   |
Copying config 6d3e4188a3 done   |
Writing manifest to image destination
Hello from Busybox!

NVIDIA GPU support

To use a GPU in a Docker container, you need to add the GPU device using the --device option.

For example, to request a GPU node with one GPU:

salloc -n 1 -G 1

After obtaining the node, you can test if the GPU device is available in a container by running:

docker run --rm --device nvidia.com/gpu=all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

If successful, the nvidia-smi command will display details about the GPU, such as model, memory usage, and driver version.

Further Reading

Supercomputer: 
Service: 

PyTorch

 PyTorch is an open source machine learning framework with GPU acceleration and deep neural networks that is based on the automatic differentiation in the Torch library of tensors.

If you installed PyTorch-nightly on Linux via pip between December 25, 2022 and December 30, 2022, please uninstall it and torchtriton immediately, and use the latest nightly binaries (newer than Dec 30th 2022). See this post page from PyTorch for detailed information. 

Publisher/Vendor/Repository and License Type

https://pytorch.org, Open source.

Availability and Restrictions

Versions

Pytorch is available on OSC Clusters. The versions currently available at OSC are:

Version Cardinal Ascend Pitzer
2.4.0 X*    
2.5.0   X  
2.7.1     X
2.8.0 X X X

 

You can use module spider pytorch to view available modules for a given machine.

Loading PyTorch from Module

A basic conda environment with PyTorch is available through the module system:

module load pytorch/2.8.0
module unload pytorch/2.8.0

The basic environment includes: pytorch, transformers, flash attention, mlflow (available with 2.8.0+), accelerate, lightning, deepspeed, diffusers, and megatron.  Examples in this documentation use version 2.8.0 but you can replace that with your target version.

Cloning PyTorch Environment

For extending the basic conda with project- or lab-specific packages, we encourage users to clone the basic environment to their project space:

module load miniconda3/24.1.2-py310
conda create --prefix /fs/project/your_project_code/your_username/your_project_name --clone /apps/pytorch/2.8.0

Then, users can install packages in the new cloned conda environment.  See HOWTO: Create and Manage Python Environments.

Installing PyTorch Locally

For alternative versions of PyTorch, users are able to create their own conda environments and install locally.  We are also available to assist with the configuration of local individual/research-group installations on all our clusters.  If you have any questions, please contact OSC Help.

Here is an example installation that was used in February 2022 to install a GPU enabled version compatible with the CUDA drivers on the clusters at that time:

Load the correct python and cuda modules:

module load miniconda3/24.1.2-py310  cuda/12.3.0
module list
Create a python environment to install pytorch into:
conda create -n pytorch
Activate the conda environment:
source activate pytorch
Install the specific version of pytorch:
pip3 install torch torchvision

PyTorch is now installed into your $HOME/local directory using the local install directory hierarchy described here and can be tested via:

module load miniconda3/24.1.2-py310 cuda/12.3.0 ; module list ; source activate pytorch
python <<EOF
import torch
    
x = torch.rand(5, 3)
print("torch.rand(5, 3) =", x)
    
print( "Is cuda available =", torch.cuda.is_available() )
exit
EOF

If testing for a GPU you will need to submit the above script as a batch job (make sure to request a GPU for the job, see Job Scripts for more info on requesting GPU)

Please refer here if you want a different version of the Pytorch.

Batch Usage

Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Scheduling Policies and Limits for more info.  In particular, Pytorch should be run on a GPU-enabled compute node.

AN EXAMPLE BATCH SCRIPT TEMPLATE

Below is an example batch script (job.sh) for using PyTorch (Slurm syntax).

Contents of job.sh

#!/bin/bash
#SBATCH --job-name=pytorch
#SBATCH --nodes=1 --ntasks-per-node=28 --gpus_per_node=1 --gpu_cmode=shared
#SBATCH --time=30:00
#SBATCH --account=yourprojectID

cd $SLURM_SUBMIT_DIR

module load miniconda3/24.1.2-py310

source activate your-local-python-environment-name

python your-pytorch-script.py

In order to run it via the batch system, submit the job.sh  file with the following command:

sbatch job.sh

GPU Usage

Jupyter Usage

PyTorch is available to be loaded as a kernel in a Jupyter notebook when running on Pitzer, Cardinal, and Ascend clusters. See HOWTO: Use Jupyter on OnDemand for details. Be sure to request GPU resources when starting your Jupyter session if you want GPU acceleration.

Further Reading

PyTorch Homepage

Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Python

Python is a high-level, multi-paradigm programming language that is both easy to learn and useful in a wide variety of applications.  Python has a large standard library as well as a large number of third-party extensions, most of which are completely free and open source. 

Availability and Restrictions

Versions

Python is available on OSC Clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Notes
3.12 X X X* Anaconda 2024.06 distribution with Python 3.12.4 (conda 24.5.0)
* Current default version

Some versions installed as an integrated package Anaconda

You can use module spider python to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Best Practices for Python Environment Management:

Utilize Miniconda3 Modules for Python Environments:Instead of relying on the default Python modules provided by OSC, leverage Miniconda3 modules for creating Python environments. Miniconda3 provides a lightweight distribution of Python and Conda, allowing for flexible environment management.
Configure Conda Channels: Before installing python packages via conda, select desired channels HOWTO: Create and Manage Python Environments based on required packages and licensing restrictions.
Maintain a Clean ~/.bashrc:It's recommended to keep your ~/.bashrc file clean and free from unnecessary scripts or Conda-related settings. This helps avoid conflicts and ensures a more predictable environment setup.
Set PYTHONNOUSERSITE before Activating Environment:Before activating a Python environment, set PYTHONNOUSERSITE=TRUE. This prevents Python from accessing and using user-installed packages located in ~/.local, ensuring a clean and isolated environment.
Deactivate Conda Environment Before Submitting Batch Jobs:Always remember to deactivate the Conda environment (source deactivate) before submitting batch jobs on the HPC system. This ensures that the job runs in a clean environment without any dependencies from the active Conda environment.

Access

Python is available for use by all OSC users, but all users are required to review and accept Anaconda, Inc. Terms of Service before accessing the software.

Publisher/Vendor/Repository and License Type

Anaconda Inc., Open source and Proprietary licenses.  See Anaconda, Inc. Terms of Service for details.

Usage

Terminal

Set-up

To load a version of the Python module, use  module load python/version. For example, use module load python/3.5 to load Python version 3.5. After the module is loaded, you can run the interpreter by using the command python. To unload the Python 3.5 module, use the command module unload python/3.5 or simply module unload python

Installed Modules

We have installed a number of Python packages and tuned them for optimal performance on our systems.  When using the Anaconda distributions of python you can run conda list to view the installed packages.

NOTE:
  • Due to architecture differences between our supercomputers, we recommend NOT installing your own packages in  ~/.local. Instead, you should install them in some other directory and set $PYTHONPATH in your default environment. For more information about installing your own Python modules, please see our HOWTO.
Environments

See the HOWTO section for more information on how to create and use python environements.

Batch

When you log into pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations (Pitzer) and Batch Limit Rules (Pitzer) for more info. 

Here is an example batch job script

#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1 
#SBATCH --time=00:01:00

module reset   # reset any loaded modules
module list    # list currently loaded modules

module load python/3.12

cp example.py $TMPDIR
cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR
    

Utilizing Python Environments Within Batch Job:

Important: When utilizing a python environment make sure to deactivate the environment before submitting the script or include source deactivate in the batch script before activating the environment.
Here is an example batch job script involving conda environment:
#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1
#SBATCH --time=00:01:00

# run to following to ensure local environment does not effect the batch job in unexpected ways

source deactivate # deactivate copy of local python environment if job submitted from within environment
module reset      # reset any loaded modules
module list       # list currently loaded modules

module load python/3.12 # load python
export PYTHONNOUSERSITE=True  #to avoid local python packages

source activate MY_ENV  # activate conda environment 


# Rest of script below

cp example.py $TMPDIR

cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR

HOW-TOs

Use Jupyter on OnDemand

OnDemand allows for use of the Jupyter interactive app. Please refer to the following page for more details:

Manage your Python packages

We highly recommend creating a local environment using Miniconda3 modules to manage Python packages for your production and research tasks. Please refer to the following how-to pages for more details:

Install packages for deep/machine learning

Advanced topics

 

Known Issues

Incorrect MPI launcher and compiler wrappers with Conda environments

Updated: March 2020
Versions Affected: Python 2.7, 3.6 & Conda 5.2
Users may encounter under-performing MPI jobs or failures of compiling MPI applications if you are using Conda from system. We found pre-installed mpich2 package in some Conda environments overrides default MPI path. The affected Conda packages are python/2.7-conda5.2 and python/3.6-conda5.2. If users experience these issues, please re-load MPI module, e.g. module load mvapich2 after setting up your Conda environment.
 

Compatibility Issues with NumPy 2.0

 

The newly released version of NumPy 2.0 includes substantial internal changes, including migrating code from C to C++. These modifications have led to significant issues with backwards compatibility, resulting in numerous breaking changes to both the Python and C APIs. As a consequence, packages built against NumPy 1.xx may encounter ImportError messages. To ensure compatibility, these packages must be rebuilt against NumPy 2.0.

Recommendation for Addressing the Issue:

  1. Follow the Migration Guide: Refer to the NumPy 2.0 Migration Guide for detailed instructions.

  2. Update Packages: Ensure all packages are updated to their latest versions.

  3. Contact Developers: Reach out to package developers for updates or compatibility information.

  4. Create a Project-Specific Environment: Set up a dedicated Python environment for your project to manage package versions effectively. Refer to the OSC documentation for guidance on using the Conda package manager.

  5. Separate Environments for Compatibility: Maintain separate Python environments for packages that are compatible with NumPy 1.x and NumPy 2.x.

Further reading

Extensive documentation of the Python programming language and software downloads can be found at the Official Python Website.  

See Also

Supercomputer: 
Service: 
Fields of Science: 

Q-Chem

Q-Chem is a general purpose ab initio electronic structure program. Its latest version emphasizes Self-Consistent Field, especially Density Functional Theory, post Hartree-Fock, and innovative algorithms for fast performance and reduced scaling calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs reasonably well within its single reference paradigm on open shell and excited state systems. The Q-Chem Home Page has additional information.

Availability and Restrictions

Versions

Q-Chem is available on the OSC clusters. These are the versions currently available:

Version Pitzer Ascend Cardinal Notes
6.3.0 X X X  
* Current default version
Note: Starting from version 5.2, the -mpi flag is required for running an MPI job, e.g., qchem -mpi -np 2. Without this flag, OpenMP will be used as the default parallelization method.
On October 12, 2023, OSC will maintain only the latest available version of Q-Chem due to the Q-Chem academic license policy. We recommend updating your job scripts if you are currently using older versions of Q-Chem. Please note that moving forward, when a new version of Q-Chem becomes available and is installed at OSC, the previous version will be automatically removed. You can use the command module avail qchem to view available Q-Chem modules for a given machine.

Access

Q-Chem is available to academic OSC users only. Please review the Q-Chem license agreement carefully before use. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Q-Chem, Inc., Commercial

Usage

For MPI jobs that request multiple nodes the job script must be run from a globally accessible working directory, e.g., project or home directories

Starting with version 5.1, QCSCRATCH is automatically set to $TMPDIR, which is removed once the job is completed. This setup helps conserve scratch space and improves job performance. If you need to save Q-Chem scratch files from a job for later use, set QCSCRATCH to a globally accessible working directory and QCLOCALSCR to $TMPDIR.

Usage

Set-up

Q-Chem usage is controlled via modules. Load one of the Q-Chem modulefiles at the command line, in your shell initialization script, or in your batch scripts. To load a particular version of Q-Chem module, use module load qchem/version. For example, use  module load qchem/6.3.0 to load Q-Chem 6.3.0.

.Examples

  • The root of the Q-Chem directory tree is $QC.
  • Example Q-Chem input files are in the $QC/samples directory

Batch Usage

When you in login environment, you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Interactive Batch Session

For an interactive batch session one can run the following command:

salloc -A <project-account> -N 1 -n 1 -t 00:20:00

which requests one core (-N 1 -n 1), for a walltime of 20 minutes (-t 00:20:00). You may adjust the numbers per your need.

Further Reading

Supercomputer: 
Service: 

QGIS

QGIS is a user friendly Open Source Geographic Information System (GIS) licensed under the GNU General Public License. QGIS is an official project of the Open Source Geospatial Foundation (OSGeo). It runs on Linux, Unix, Mac OSX, Windows and Android and supports numerous vector, raster, and database formats and functionalities.

Availability and Restrictions

Versions

The following versions of QGIS are available on OSC clusters:

Version Pitzer Note
3.22.8 X SAGA 7.9.1 available

Access

QGIS is available to all OSC users via OnDemand QGIS app. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

GNU General Public License.

Further Reading

Supercomputer: 
Service: 

Quantum ESPRESSO

Quantum ESPRESSO (QE) is a program package for ab-initio molecular dynamics (MD) simulations and electronic structure calculations.  It is based on density-functional theory, plane waves, and pseudopotentials.

Availability and Restrictions

Versions

The following versions are available on OSC systems:

Version Pitzer Ascend Cardinal Note
6.7 X      
7.3.1 X X X*  
7.4.1 X X X  
* Current default version

You can use module spider quantum-espresso to view available modules.  To select a particular software version, use module load quantum-espresso/version. For example, use module load quantum-espresso/7.3.1 to load Quantum Espresso version 7.3.1; and after loading use module help quantum-espresso/7.3.1 to view details, such as, installed packages and  compiler prerequisites; some versions require specific prerequisite modules, and such details may be obtained with the command module spider quantum-espresso/version.  Feel free to contact OSC Help if you need other versions for your work.

Access

Quantum ESPRESSO is open source and available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

http://www.quantum-espresso.org, Open source

Usage

Set-up

You can configure your environment for the usage of Quantum ESPRESSO by running the following command:

module load quantum-espresso

In the case of multiple compiled versions load the appropriate compiler first, e.g., on Cardinal to select the most recently compiled QE 7.3.1 version use the following commands:

module load intel/2021.10.0 openmpi/5.0.2
module load quantum-espresso/7.3.1

Batch Usage

Sample batch scripts and input files are available here:

~srb/workshops/compchem/espresso/

Performance

The performance results were obtained by running the GRIR443 benchmark using Quantum ESPRESSO version 7.3.1.

Cluster # CPUs Build Dependencies CPU Time Wall Time
Cardinal 96 intel/2021.10.0 openmpi/5.0.2 12m43s 13m4s
Ascend (nextgen) 120 intel/2021.10.0 openmpi/5.0.2 23m56s 24m22s
Pitzer 48 intel/2021.10.0 openmpi/5.0.2 27m16s 28m39s
Pitzer (RHEL 7) 48 intel/19.0.5 mvapich2/2.3.6 27m59s 29m31s

Further Reading

See Also

Supercomputer: 
Service: 

R and Rstudio

R is a language and environment for statistical computing and graphics. It is an integrated suite of software facilities for data manipulation, calculation, and graphical display. It includes

  • an effective data handling and storage facility,
  • a suite of operators for calculations on arrays, in particular matrices,
  • a large, coherent, integrated collection of intermediate tools for data analysis,
  • graphical facilities for data analysis and display either on-screen or on hardcopy, and
  • a well-developed, simple and effective programming language which includes conditionals, loops, user-defined recursive functions and input, and output facilities

More information can be found here.

Availability and Restrictions

Versions

The following versions of R are available on OSC systems: 

Version Pitzer Ascend Cardinal
4.4.0 X X X*

 

R/4.4.0 l is compiled with gcc/12.3.0. To load R/4.4.0, please load the gcc/12.3.0 module first.
* Current default version
** The user state directory (session data)  is stored at ~/.local/share/rstudio for the latest RStudio that we have deployed with R/4.1.0. It is located at ~/.rstudio for older versions.  Users would need to delete session data from ~/.local/share/rstudio for R/4.1.0 and ~/.rstudio for older versions to clear workspace history.

Known Issue

There's a known issue loading modules in RStudio's environment after changing versions or clusters.

If you have issues using modules in the RConsole - try these remedies

  • restarting the terminal
  • restarting the RConole
  • logging out of the RStudio session and logging back in.
  • remove your ~/.local/share/rstudio

You can use module avail R to view available modules and module spider R/version to show how to load the module for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

R is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

R Foundation, Open source

Usage

R software can be launched two different ways; through Rstudio on OSC OnDemand and through the terminal.

Rstudio

In order to access Rstudio and OSC R workshop materials, please visit here.

Terminal Acess

In order to configure your environment for R, please run the following command:

module load gcc/version R/version
#for example,
module load gcc/12.3.0 R/4.4.0

R/4.4.0 and onwards versions use gcc compiler. Loading R/4.4.0 requires the dependency gcc/12.3.0 to also be loaded.

Using R

Once your environment is configured, R can be started simply by entering the following command:

R

For a listing of command line options, run:

R --help

Running R interactively on a login node for extended computations is not recommended and may violate OSC usage policy. Users can either request compute nodes to run R interactively or run R in batch.

Running R interactively on terminal:

Request compute node or nodes if running parallel R as,

sinteractive -A <project-account> -N 1 -n 28 -t 01:00:00 

When the compute node is ready, launch R by loading modules

module load  gcc/12.3.0 R/4.4.0
R

Batch Usage

 Reference the example batch script below. This script requests one full node on the Cardinal cluster for 1 hour of wall time.

#!/bin/bash
#SBATCH --job-name R_ExampleJob
#SBATCH --nodes=1 --ntasks-per-node=48
#SBATCH --time=01:00:00
#SBATCH --account <your_project_id>

module load gcc/12.3.0    
module load R/4.4.0
    
cp in.dat test.R $TMPDIR
cd $TMPDIR
    
R CMD BATCH test.R test.Rout
    
cp test.Rout $SLURM_SUBMIT_DIR

HOWTO: Install Local R Packages

R comes with a single library  $R_HOME/library which contains the standard and recommended packages. This is usually in a system location. 

Users can check the library path as follows after launching an R session;

> .libPaths()
[1] "/users/PZS0680/soottikkal/R/x86_64-pc-linux-gnu-library/3.6"
[2] "/usr/local/R/gnu/9.1/3.6.3/site/pkgs"
[3] "/usr/local/R/gnu/9.1/3.6.3/lib64/R/library"

Users can check the list of available packages as follows;

>installed.packages()

To install local R packages, use install.package() command. For example,

>install.packages("lattice")

 For the first time local installation, it will give a warning as follows:

Installing package into ‘/usr/local/R/gnu/9.1/3.6.3/site/pkgs’
(as ‘lib’ is unspecified)
Warning in install.packages("lattice") :
'lib = "/usr/local/R/gnu/9.1/3.6.3/site/pkgs"' is not writable
Would you like to use a personal library instead? (yes/No/cancel)

Answer , and it will create the directory and install the package there.

Installing Packages from GitHub

Users can install R packages directly from Github using devtools package as follows

>install.packages("devtools")
>devtools::install_github("author/package")

If you get errors related to the R XML package, see the Troubleshooting Issues section.

Installing Packages from Bioconductor

Users can install R packages directly from Bioconductor using BiocManager.

>install.packages("BiocManager")
>BiocManager::install(c("GenomicRanges", "Organism.dplyr"))
    

R packages with external dependencies 

When installing R packages with external dependencies, users may need to import appropriate libraries into R. One of the frequently requested R packages is sf which needs geos, gdal and PROJ libraries (For more. We have a few versions of those packages installed and they can be loaded as modules. Another relativey common external dependency is gsl. To see what versions of modules are available, run the command module spider from the command line. For example, to see what version of gsl is available to load, run module spider gsl. The output will look something like this:

------------------------------------------------------------------------
gsl: gsl/2.7.1
------------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "gsl/2.7.1" module is available to load.

      gcc/12.3.0
      intel/2021.10.0

According to this output, we should run module load gcc/12.3.0 gsl/2.7.1 or module load intel/2021.10.0 gsl/2.7.1. You can also load modules directly from the R terminal in Rstudio:

> source(file.path(Sys.getenv("LMOD_PKG"), "init/R"))
> module("load", "geos/version")

For example, to load the module geos/3.12.0, you would run module("load", "geos/3.12.0"). You can check if an external pacakge is available

> module("avail", "geos")

When modules of external libs are not available, users can install those and link libraries to the R environment. Suppose you have locally installed gdal/3.3.1, and proj/9.2.1 at the path /users/<account-number>/<username>/local. Here is an example of how to install the sf package on Cardinal without modules.

# Update LD_LIBRARY_PATH to include user-installed libraries.
>old_ld_path <- Sys.getenv("LD_LIBRARY_PATH")
>Sys.setenv(LD_LIBRARY_PATH = paste(old_ld_path, "/users/<account-number>/<username>/local/gdal/3.3.1/lib", "/users/<account-number>/<username>/local/proj/9.2.1","/users/<account-number>/<username>/local/geos/3.9.1/lib",sep=":"))

>Sys.setenv("PKG_CONFIG_PATH"="/users/<account-number>/<username>/local/proj/9.2.1/lib/pkgconfig")
>Sys.setenv("GDAL_DATA"="/users/<account-number>/<username>/local/gdal/3.3.1/share/gdal")

>install.packages("sf", configure.args=c("--with-gdal-config=/users/<account-number>/<username>/local/gdal/3.3.1/bin/gdal-config","--with-proj-include=/users/<account-number>/<username>/local/proj/8.1.0/include","--with-proj-lib=/users/<account-number>/<username>/local/proj/9.2.1/lib"),INSTALL_opts="--no-test-load")

>dyn.load("/users/<account-number>/<username>/local/gdal/3.3.1/lib64/libgdal.so")
>dyn.load("/users/<account-number>/<username>/local/proj/9.2.1/lib64/libproj.so", local=FALSE)
>library(sf)

Please note that every time before loading sf package, you have to execute the dyn.load of both libraries listed above. 

renv: Package Manager

if you are using R for multiple projects, OSC recommendsrenv, an R dependency manager for R package management. Please see more information here.

The renv package helps you create reproducible environments for your R projects. Use renv to make your R projects more:

  • Isolated: Each project gets its own library of R packages, so you can feel free to upgrade and change package versions in one project without worrying about breaking your other projects.

  • Portable: Because renv captures the state of your R packages within a lockfile, you can more easily share and collaborate on projects with others, and ensure that everyone is working from a common base.

  • Reproducible: Use renv::snapshot() to save the state of your R library to the lockfile renv.lock. You can later use renv::restore() to restore your R library exactly as specified in the lockfile.

Users can install renv package as follows;

>install.packages("renv")

The core essence of the renv workflow is fairly simple:

  1. After launching R, go to your project directory using R command setwd and initiate renv:

    setwd("your/project/path")
    renv::init()

    This function forks the state of your default R libraries into a project-local library. A project-local .Rprofile is created (or amended), which is then used by new R sessions to automatically initialize renv and ensure the project-local library is used. 

    Work in your project as usual, installing and upgrading R packages as required as your project evolves.

  2. Use renv::snapshot() to save the state of your project library. The project state will be serialized into a file called renv.lock under your project path.

  3. Use renv::restore() to restore your project library from the state of your previously-created lockfile renv.lock.

In short: use renv::init() to initialize your project library, and use renv::snapshot() / renv::restore() to save and load the state of your library.

After your project has been initialized, you can work within the project as before, but without fear that installing or upgrading packages could affect other projects on your system.

Global Cache

One of renv’s primary features is the use of a global package cache, which is shared across all projects using renvWhen using renv the packages from various projects are installed to the global cache. The individual project library is instead formed as a directory of symlinks  into the renv global package cache. Hence, while each renv project is isolated from other projects on your system, they can still re-use the same installed packages as required. By default, global Cache of renv is located ~/.local/share/renvUser can change the global cache location using RENV_PATHS_CACHE variable. Please see more information here.

Please note that renv does not load packages from site location (add-on packages installed by OSC) to the rsession. Users will have access to the base R packages only when using renv. All other packages required for the project should be installed by the user.

Version Control with renv

If you would like to version control your project, you can utilize git versioning of renv.lock file. First, initiate git for your project directory on a terminal

git init

Continue working on your R project by launching R, installing packages, saving snapshot using renv::snapshot()command. Please note that renv::snapshot() will only save packages that are used in the current project. To capture all packages within the active R libraries in the lockfile, please see the type option. 

>renv::snapshot(type="simple")

If you’re using a version control system with your project, then as you call renv::snapshot() and later commit new lockfiles to your repository, you may find it necessary later to recover older versions of your lockfiles. renv provides the functions renv::history()to list previous revisions of your lockfile, and renv::revert() to recover these older lockfiles.

If you are using renvpackage for the first time, it is recommended that you check R startup files in your $HOME such as .Rprofile and .Renviron and remove any project-specific settings from these files. Please also make sure you do not have any project-specific settings in ~/.R/Makevars.

A Simple Example

First, you need to load the module for R and fire up R session

module load R/4.4.0
R

Then set the working directory and initiate renv

setwd("your/project/path")
renv::init()

Let's install a package called  lattice,  and save the snapshot to the renv.lock

renv::install("lattice")
renv::snapshot(type="simple")

The latticepackage will be installed in global cache of renv and symlink will be saved in renv under the project path.

Restore a Project

Use renv::restore() to restore a project's dependencies from a lockfile, as previously generated by snapshot(). Let's remove the lattice package.

renv::remove("lattice")

Now let's restore the project from the previously saved snapshot so that the lattice package is restored.

renv::restore()
library(lattice)

Collaborating with renv

When using renv, the packages used in your project will be recorded into a lockfile, renv.lock. Because renv.lock records the exact versions of R packages used within a project, if you share that file with your collaborators, they will be able to use renv::restore() to install exactly the same R packages as recorded in the lockfile. Please find more information here.

Parallel R

Please set the environment variables OMP_NUM_THREADS and MKL_NUM_THREADS to 1 in your job scripts. This adjustment helps avoid additional internal parallel processing by libraries such as OpenMP and MKL, which can otherwise conflict with parallelism set by R’s parallel processing packages.

R provides a number of methods for parallel processing of the code. Multiple cores and nodes available on OSC clusters can be effectively deployed to run many computations in R faster through parallelism.

Consider this example, where we use a function that will generate values sampled from a normal distribution and sum the vector of those results; every call to the function is a separate simulation.

    myProc <- function(size=1000000) {
      # Load a large vector
      vec <- rnorm(size)
      # Now sum the vec values
      return(sum(vec))
    }

Serial execution with loop

Let’s first create a serial version of R code to run myProc() 100x on Pitzer:

    tick <- proc.time()
    for(i in 1:100) {
      myProc()
    }
    tock <- proc.time() - tick
    tock
    ##    user  system elapsed
    ##   6.437   0.199   6.637

Here, we execute each trial sequentially, utilizing only one of our 28 processors on this machine. In order to apply parallelism, we need to create multiple tasks that can be dispatched to different cores. Using apply() family of R function, we can create multiple tasks. We can rewrite the above code  to use apply(), which applies a function to each of the members of a list (in this case the trials we want to run):

    tick <- proc.time()
    result <- lapply(1:100, function(i) myProc())
    tock <-proc.time() - tick
    tock
    ##    user  system elapsed
    ##   6.346   0.152   6.498

parallel package

The  parallellibrary can be used to dispatch tasks to different cores. The parallel::mclapply function can distributes the tasks to multiple processors.

    library(parallel)
    cores <- system("nproc", intern=TRUE)
    tick <- proc.time()
    result <- mclapply(1:100, function(i) myProc(), mc.cores=cores)
    tock <- proc.time() - tick
    tock
    ##    user  system elapsed
    ##   8.653   0.457   0.382

foreach package

The foreach package provides a  looping construct for executing R code repeatedly. It uses the sequential %do% operator to indicate an expression to run.

    library(foreach)
    tick <- proc.time()
    result <-foreach(i=1:100) %do% {
       myProc()
    }
    tock <- proc.time() - tick
    tock
    ##    user  system elapsed
    ##   6.420   0.018   6.439

doParallel package

foreach supports a parallelizable operator %dopar% from the doParallel package. This allows each iteration through the loop to use different cores.

    library(doParallel, quiet = TRUE)
    library(foreach)
    cl <- makeCluster(28)
    registerDoParallel(cl)
    
    tick <- proc.time()
    result <- foreach(i=1:100, .combine=c) %dopar% {
        myProc()
    }
    tock <- proc.time() - tick
    tock
    invisible(stopCluster(cl))
    detachDoParallel()
    
    ##    user  system elapsed
    ##   0.085   0.013   0.446
    

Rmpi package

Rmpi package allows to parallelize R code across multiple nodes. Rmpi provides an interface necessary to use MPI for parallel computing using R. This allows each iteration through the loop to use different cores on different nodes. Rmpijobs cannot be run with RStudio at OSC currently, instead users can submit Rmpi jobs through terminal App. R uses openmpi as MPI interface therefore users would need to load openmpi module before installing or using Rmpi. Rmpi is installed at central location for R versions prior to 4.2.1. If it is not availbe, users can install it as follows

Rmpi Installation

   # Get source code of desired version of RMpi
wget https://cran.r-project.org/src/contrib/Rmpi_0.7-2.tar.gz

# Load modules
ml openmpi/5.0.2 gcc/12.3.0 R/4.4.0

# Install RMpi
R CMD INSTALL --configure-vars="CPPFLAGS=-I$MPI_HOME/include LDFLAGS='-L$MPI_HOME/lib'" --configure-args="--with-Rmpi-include=$MPI_HOME/include --with-Rmpi-libpath=$MPI_HOME/lib --with-Rmpi-type=OPENMPI" Rmpi_0.7-2.tar.gz

# Test loading
library(Rmpi)

   

Please make sure that $MPI_HOME is defined after loading openmpi module. Newer versions of openmpi module has $OPENMPI_HOME instead of $MPI_HOME. So you would need to replace $MPI_HOME with $OPENMPI_HOME for those versions of openmpi.

Above example code can be rewritten to utilize multiple nodes with Rmpias follows;

    library(Rmpi)
    library(snow)
    workers <- as.numeric(Sys.getenv(c("PBS_NP")))-1
    cl <- makeCluster(workers, type="MPI") # MPI tasks to use
    clusterExport(cl, list('myProc'))
    tick <- proc.time()
    result <- clusterApply(cl, 1:100, function(i) myProc())
    write.table(result, file = "foo.csv", sep = ",")
    tock <- proc.time() - tick
    tock

Batch script for job submission is as follows;

    #!/bin/bash
    #SBATCH --time=10:00
    #SBATCH --nodes=2 --ntasks-per-node=28
    #SBATCH --account=<project-account>
    #SBATCH --export=ALL,OMP_NUM_THREADS=1,MKL_NUM_THREADS=1
    
    module reset
    module load openmpi/5.0.2 gcc/12.3.0 R/4.4.0
    
    # parallel R: submit job with one MPI master
    mpirun -np 1 R --slave < Rmpi.R

pbdMPI package

pbdMPI is an improved version of the Rmpi package that provides efficient interface to MPI by utilizing S4 classes and methods with a focus on Single Program/Multiple Data ('SPMD') parallel programming style, which is intended for batch parallel execution. This means that all processes (ranks) run the same code independently. pbdMPI also uses OpenMPI as an MPI interface. 

Installation of pbdMPI

Users can download latest version of pbdMPI from CRAN https://cran.r-project.org/web/packages/pbdMPI/index.html and install it as follows:

wget https://cran.r-project.org/src/contrib/pbdMPI_0.5-3.tar.gz
ml gcc/12.3.0
ml R/4.4.0
ml openmpi/5.0.2
R CMD INSTALL pbdMPI_0.5-3.tar.gz

Examples

Example of a matrix calculation using pbdMPI:

# Load the pbdMPI package
library(pbdMPI, quietly = TRUE)

# Initialize MPI environment
init()

# Each rank creates 2 matrices with random data
matrix_size = 5
set.seed(100 + comm.rank())
A <- matrix(rnorm(matrix_size^2), nrow = matrix_size)
B <- matrix(rnorm(matrix_size^2), nrow = matrix_size)

# Multiply the matrices
C <- A %*% B

# Gather all C matrices to rank 0
gathered_C <- gather(C, rank.dest = 0)

# On rank 0, compute the global sum of all 
if (comm.rank() == 0) {
  global_sum <- Reduce("+", gathered_C)
  cat("Global sum of all C matrices:\n")
  print(global_sum)
}

finalize()

An example batch job submission script is as follows:

#!/bin/bash
#SBATCH --time=00:10:00
#SBATCH --nodes=2 --ntasks-per-node=4
#SBATCH --account=<project-account>
#SBATCH --export=ALL,OMP_NUM_THREADS=1,MKL_NUM_THREADS=1

module reset
module load gcc/12.3.0 R/4.4.0 openmpi/5.0.2

mpirun Rscript pbdMPI-script.R

Note that one copy of this script will be run for each node, so the total number of tasks will affect the total number of matrices operations computed. In this example with 8 total tasks, 16 total matrices will be created (2 per task).

Here are additional resources that demonstrate how to use pbdMPI:

https://cran.r-project.org/web/packages/pbdMPI/pbdMPI.pdf

http://hpcf-files.umbc.edu/research/papers/pbdRtara2013.pdf

Paralell R jobs can be monitored in Grafana by visiting the link outputted from the command job-dashboard-link.py <jobid>

R Batchtools

The R package, batchtools provides a parallel implementation of Map for high-performance computing systems managed by schedulers Slurm on OSC system. Please find more info here https://github.com/mllg/batchtools.

Users would need two files slurm.tmpl and .batch.conf.R

Slurm.tmpl is provided below. Please change "your project_ID".

    #!/bin/bash -l
    ## Job Resource Interface Definition
    ## ntasks [integer(1)]:       Number of required tasks,
    ##                            Set larger than 1 if you want to further parallelize
    ##                            with MPI within your job.
    ## ncpus [integer(1)]:        Number of required cpus per task,
    ##                            Set larger than 1 if you want to further parallelize
    ##                            with multicore/parallel within each task.
    ## walltime [integer(1)]:     Walltime for this job, in seconds.
    ##                            Must be at least 60 seconds.
    ## memory   [integer(1)]:     Memory in megabytes for each cpu.
    ##                            Must be at least 100 (when I tried lower values my
    ##                            jobs did not start at all).
    ## Default resources can be set in your .batchtools.conf.R by defining the variable
    ## 'default.resources' as a named list.
    
    <%
    # relative paths are not handled well by Slurm
    log.file = fs::path_expand(log.file)
    -%>
    
    #SBATCH --job-name=<%= job.name %>
    #SBATCH --output=<%= log.file %>
    #SBATCH --error=<%= log.file %>
    #SBATCH --time=<%= ceiling(resources$walltime / 60) %>
    #SBATCH --ntasks=1
    #SBATCH --cpus-per-task=<%= resources$ncpus %>
    #SBATCH --mem-per-cpu=<%= resources$memory %>
    #SBATCH --account=your_project_id
    <%= if (!is.null(resources$partition)) sprintf(paste0("#SBATCH --partition='", resources$partition, "'")) %>
    <%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>
    
    
    ## Initialize work environment like
    ## source /etc/profile
    ## module add ...
    
    module add  R/4.0.2-gnu9.1
    
    ## Export value of DEBUGME environemnt var to slave
    export DEBUGME=<%= Sys.getenv("DEBUGME") %>
    <%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
    <%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
    <%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
    
    
    ## Run R:
    ## we merge R output with stdout from SLURM, which gets then logged via --output option
    
    Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

.batch.conf.R is provided below.

    cluster.functions = makeClusterFunctionsSlurm(template="path/to/slurm.tmpl")

A test example is provided below. Assuming the current working directory has both slurm.tmpl and .batch.conf.R files.

    ml gcc/12.3.0 R/4.4.0
    R
    
    >install.packages("batchtools")
    >library(batchtools)
    >myFct <- function(x) {
    result <- cbind(iris[x, 1:4,],
    Node=system("hostname", intern=TRUE),
    Rversion=paste(R.Version()[6:7], collapse="."))}
    
    >reg <- makeRegistry(file.dir="myregdir", conf.file=".batchtools.conf.R")
    >Njobs <- 1:4 # Define number of jobs (here 4)
    >ids <- batchMap(fun=myFct, x=Njobs)
    >done <- submitJobs(ids, reg=reg, resources=list( walltime=60, ntasks=1, ncpus=1, memory=1024))
    >waitForJobs()
    >getStatus() # Summarize job
    
    

Profiling R code

Profiling R code helps to optimize the code by identifying bottlenecks and improve its performance. There are a number of tools that can be used to profile R code.

Grafana:

OSC jobs can be monitored for CPU and memory usage using grafana.  If your job is in running status, you can get grafana metrics as follows. After log in to OSC OnDemand, select Jobs from the top tabs, then select Active Jobs and then Job that you are interested to profile. You will see grafana metrics at the bottom of the page and you can click on detailed metrics to access more information about your job at grafana.

Screen Shot of grafana metrics

Rprof:

R’s built-in tool,Rprof function can be used to profile R expressions and the summaryRprof function to summarize the result. More information can be found here.

Here is an example of profiling R code with Rprofe for data analysis on Faithful data.

Rprof("Rprof-out.prof",memory.profiling=TRUE, line.profiling=TRUE)
data(faithful)
summary(faithful)
plot(faithful)
Rprof(NULL)    

To analyze profiled data, runsummaryRprof on Rprof-out.prof

summaryRprof("Rprof-out.prof")

You can read more about summaryRprofhere

Profvis:

 It provides an interactive graphical interface for visualizing data from Rprof.

library(profvis)
profvis({
    data(faithful)
    summary(faithful)
    plot(faithful)
},prof_output="profvis-out.prof")

If you are running the R code on Rstudio, it will automatically open up the visualization for the profiled data. More info can be found here.

Using Rstudio for classroom  

OSC provides an isolated and custom R environment for each classroom project that requires Rstudio. More information can be found here.

Further Reading

Troubleshooting issues  

Check .bashrc

If you're encountering difficulties launching the RStudio App on-demand or errors with installing packages, the first step is to review your ~/.bashrc file. Check for custom configurations and any conda/python related lines. Consider commenting out these configurations and attempting to launch the app or re-install the package.

R session taking too long to initialize

If your R session is taking too long to initialize, it might be due to issues from a previous session. First, make sure no Rstudio jobs are running. Then restore R to a fresh session by removing the previous state stored at

~/.local/share/rstudio (~/.rstudio for <R/4.1)

mv ~/.local/share/rstudio ~/.local/share/rstudio.backup

Common Problem Packages

Several packages are known to have problems installing.

XML

For R XML, the libxml2 library must be preloaded. This is sometimes also needed for packages that depend on the XML package, such as rtracklayer.

> Sys.setenv("LD_PRELOAD"="/lib64/libxml2.so")
> dyn.load("/lib64/libxml2.so") 
> install.packages("XML")

sf

For sf: proj and gdal modules must be loaded. Follow the instructions in the R packages with external dependencies section to see which versions of these modules are available and load them. You must also call dyn.load on their libraries. To find out the correct path, run the command module show module/version. For example, run module show proj/9.2.1, if this is the version available. The output should include a line that looks like this:

> setenv("PROJ_HOME","/apps/spack/0.21/pitzer/linux-rhel9-skylake/proj/gcc/12.3.0/9.2.1-buhooyr")

The dyn.load command is the path here concatenated with /lib/libproj.so. Follow the same steps for gdal. Here is full example of the install steps on Pitzer:

> source(file.path(Sys.getenv("LMOD_PKG"), "init/R"))
> module("load", "proj/9.2.1") 
> module("load", "gdal/3.7.3") 

> dyn.load("/apps/spack/0.21/pitzer/linux-rhel9-skylake/proj/gcc/12.3.0/9.2.1-buhooyr/lib64/libproj.so")
> dyn.load("/apps/spack/0.21/pitzer/linux-rhel9-skylake/gdal/gcc/12.3.0/3.7.3-wmnbnyd/lib64/libgdal.so")

> install.packages("sf")
> library("sf")

Now you can install other packages that depend on sf normally.

This is an example of the stars package installation, which has a dependency of sf package.

>install.packages("stars")
>library(stars) 

rJava

For rJava, please run the following command before attempting installation:
Last updated 9/25/25

Pitzer:

> Sys.setenv(LDFLAGS = "-L/apps/spack/0.21/pitzer/linux-rhel9-skylake/libiconv/gcc/12.3.0/1.17-bcgrlj2/lib)
Ascend:
> Sys.setenv(LDFLAGS = "-L/apps/spack/0.21/ascend/linux-rhel9-zen2/libiconv/gcc/12.3.0/1.17-wifr2il/lib)
Cardinal
> Sys.setenv(LDFLAGS = "-L/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/libiconv/gcc/12.3.0/1.17-fxsid3a/lib

 

Further Reading

See Also

Supercomputer: 
Service: 
Fields of Science: 

RELION

RELION (REgularised LIkelihood OptimisatioN) is a stand-alone computer program for the refinement of 3D reconstructions or 2D class averages in electron cryo-microscopy. 

Availability and Restrictions

Versions

RELION is available on the OSC clusters. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Note
4.0.1 X     Built with CUDA 11.8 and OpenMPI 5.0
5.0.0   X   Built with CUDA 11.8 and OpenMPI 5.0
5.0.1 X X X Built with CUDA 12 and OpenMPI 5.0

You can use module spider relion  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Available third-party packages

Cluster RELION CTFFIND* MotionCor2** GCTF ResMap
Pitzer 4.0.1 4.1.14 1.4.5    
  5.0.1 4.1.14      
Ascend 5.0 4.1.14 1.4.5    
  5.0.1 4.1.14      
Cardinal 5.0.1 4.1.14      

* To find the full path of CTFFIND, type echo $RELION_CTFFIND_EXECUTABLE after setting up RELION.

** Starting with version 5.0.1, we have removed the built-in MotionCor2. If you need to use it with RELION, please visit https://emcore.ucsf.edu/ucsf-software to download a compatible version.

Access

RELION is available to all OSC users.

Publisher/Vendor/Repository and License Type

MRC Lab of Molecular Biology, Open source

Usage

Set Up

To find which RELION versions are available on a cluster, run:

module spider relion

Follow the instructions provided in the output to see more details and to load a specific version of RELION.

For example, to prepare the environment for RELION 5.0.1, use the following commands:

module load intel/2021.10.0
module load openmpi/5.0.2
module load relion/5.0.1

RELION Python modules

To use the optional Python-based tools available in RELION, such as Topaz, Blush, ModelAngelo, and DynaMight, you need to load the corresponding Python support modules. After loading the main RELION module, add the following:

module load relion-python/5.0.1

Run Jobs via Queue Submission

In the RELION GUI, several types of jobs can be executed through queue submission to run on another computing nodes with more resources. To enable this feature, set "Submit to queue?" to "Yes" under the "Running" tab.

Screenshot 2025-10-24 at 5.58.36 PM.png

This will activate all queue job options. Please note that starting from RELION 5.0.1, the queue job options have been redesigned to improve long-term usability and to better align with our Slurm configuration, as shown above.

For the first two options at the top:

  • Number of MPI procs: Based on our understanding, this option only affects non-queue jobs when the Submit to queue option is disabled. It determines the number of MPI processes used for a local job.
  • Number of threads: This value must be consistent with the setting for Number of threads per task. It specifies the number passed to the --j option, which controls the threading level for each RELION module.

If you are reusing an existing project, the default values for these options may be replaced by settings from a previous configuration. Additionally, if the project was created with RELION 5.0 or an earlier version, some values may not match the current options because the number and layout of queue job parameters can differ between versions. If you are unsure about the purpose of these options, please apply the default values listed in the Queue Job Options section below.

We apologize for any inconvenience this may cause.

Queue Job Options

Option Default Note
Queue name None This option is not used.
Queue submit command sbatch Defines the submit command. You can use the default value only.
Number of nodes 1 Specifies the number of nodes. Sets --nodes in the job.
Number of tasks per node 1 Specifies the number of MPI processes per node. Sets --ntasks-per-node in the job.
Number of threads per task 1 Specifies the number of threads per MPI process. Sets --cpus-per-task in the job. Must larger or equal to the Number of threads option.
Set up GPU job None Leave blank for CPU-only jobs. To enable GPU, type --gpus-per-node=N, where N is the number of GPUs needed per node.
Wall time limit 1:00:00 Specifies the job wall time. Sets --time in the job.
Project account None Specifies the project account. Sets --account in the job. This is mandatory.
Event notification FAIL Specifies the type of event to send email notifications for. Sets --mail-type in the job.
Additional SBATCH directives None Adds extra SLURM directives.
Add extra MPI task No Select Yes if an extra MPI process is needed on the head node.

The last option, "Standard submission script," has the default value

/users/PZS0645/support/share/apps/relion/osc_slurm_relion5.sh

This script serves as the job submission template for RELION version 5.0.1 and later. You can copy it to any preferred location and modify it for convenience.

Further Reading

Supercomputer: 
Service: 

Rosetta

Rosetta

 

Rosetta is a software suite that includes algorithms for computational modeling and analysis of protein structures. It has enabled notable scientific advances in computational biology, including de novo protein design, enzyme design, ligand docking, and structure prediction of biological macromolecules and macromolecular complexes.

 

Availability and Restrictions

Versions

The Rosetta suite is available on Pitzer and Cardinal. The versions currently available at OSC are:

 

Version Pitzer Ascend Cardinal
3.12 X X                       X*

* Current default version

You can use  module spider rosetta to view available modules for a given machine. Feel free to contact  oschelp@osc.edu if you need other versions for your work.

Access for Academic Users 

Rosetta is available to academic OSC users. Please review the license agreement carefully before use. If you have any questions, please contact oschelp@osc.edu.

Publisher/Vendor/Repository and License Type

Rosetta, Non-Commercial

Usage

Usage on Pitzer

Further Reading

 

 

Fields of Science: 

Bioinformatics & Biology

Chemical Engineering & Chemistry

Materials

 

Supercomputer: 
Service: 
Fields of Science: 

Ruby

Ruby is a dynamic, open source programming language with a focus on simplicity and productivity.

Availability and Restrictions

Versions

The following versions of ruby are available on OSC clusters:

Version Pitzer Ascend Cardinal
3.1.5 X# X# X#
3.3.6   X  
# System version

You can use module spider ruby to view available modules. Feel free to contact OSC Help if you need other versions for your work.

Access

Ruby is available to all OSC users. If you have any questions, please contact OSC Help.

Usage

Usage on Ascend

Set-up

To configure your environment for use of ruby, run the following command:  module load ruby/3.3.6


Further Reading

Tag: 
Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Rust

Rust is a general-purpose programming language with an emphasis on performance, type safety, and concurrency. It enforces memory safety without a traditional garbage collector, preventing data races and memory safety errors via the "borrow checker". The Rust module provides rustc and cargo.

Availability and Restrictions

Versions

The following versions of Rust are available on OSC clusters:

Version Cardinal
1.81.0 X*
* Current default version

You can use module spider rust to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Rust is available to all OSC users. If you have any questions, please contact OSC Help.

Usage

Usage on Cardinal

Set-up

To configure your environment for use of Rust, run the following command:  module load rust. The default version will be loaded. This will load cargo and rustc into the environment


Further Reading

Tag: 
Supercomputer: 
Service: 
Fields of Science: 

SAMtools

SAM format is a generic format for storing large nucleotide sequence alignments. SAMtools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format.

Availability and Restrictions

The following versions of SAMtools are available on OSC clusters:

Version Pitzer Ascend Cardinal
1.17 X X X
1.21 X X X*
* Current default version

You can use  module spider samtools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

SAMtools is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Genome Research Ltd., Open source

Usage

Set-up

To configure your environment for use of SAMtools, run the following command:   module load samtools/version. For example, use  module load samtools/1.17 to load SAMtools 1.17.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

SRA Toolkit

The Sequence Read Archive (SRA Toolkit) stores raw sequence data from "next-generation" sequencing technologies including 454, IonTorrent, Illumina, SOLiD, Helicos and Complete Genomics. In addition to raw sequence data, SRA now stores alignment information in the form of read placements on a reference sequence. Use SRA Toolkit tools to directly operate on SRA runs.

Availability and Restrictions

The following versions of SRA Toolkit are available on OSC clusters:

Version Pitzer Cardinal Note
3.0.2 X X*  
* Current default version
** NCBI now uses cloud-style object stores. To access SRA cloud data, use version 2.10 or later and provide your AWS or GCP access credentials (recommended) to vdb-config. For more information, see https://github.com/ncbi/sra-tools/wiki/04.-Cloud-Credentials.

You can use  module spider sratoolkit to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

SRA Toolkit is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

National Center for Biotechnology Information, Freeware

Usage

Usage on Pitzer

Set-up

To configure your environment for use of SRA Toolkit, run the following command: module load sratoolkit/version. For example, use module load sratoolkit/2.11.2 to load SRA Toolkit 2.11.2

Download SRA Data

NCBI now uses cloud-style object stores. To access SRA cloud data, use version 2.10 or later and provide your AWS or GCP access credentials (recommended) to vdb-config. For more information, see https://github.com/ncbi/sra-tools/wiki/04.-Cloud-Credentials.

Set up the credentials (recommended)

Once you have obtained an AWS or GCP credential file, you can set the credentials by following these steps:

module load sratoolkit/3.0.2
vdb-config --report-cloud-identity yes 

# For GCP credentials
vdb-config --set-gcp-credentials /path/to/gcp/creddential/file

# For AWS credentials
vdb-config --set-aws-credentials /path/to/aws/creddential/file
Each version of the toolkit comes with its own set of configuration options. To modify the defaults, run vdb-config -i to access the interactive configuration. For additional information, please visit the following link: https://github.com/ncbi/sra-tools/wiki/03.-Quick-Toolkit-Configuration.

You can now download SRA data using prefetch 

prefetch SRR390728

The default download path is located in your home directory at ~/ncbi. For instance, if you're looking for the SRA file SRR390728.sra, you can find it at ~/ncbi/sra, and the resource files can be found at ~/ncbi/refseq. You can use srapath to verify if the SRA accession is accessible in the download path

$ srapath SRR390728
/users/PAS1234/johndoe/ncbi/sra/sra/SRR390728.sra

You can now run other SRA tools, such as fastq-dump, on computing nodes. Here is an example job script:

#!/bin/bash
#SBATCH --job-name use_fastq_dump
#SBATCH --time=0:10:0
#SBATCH --ntasks-per-node=1

module load sratoolkit/3.0.2
module list
fastq-dump -X 5 -Z SRR390728

Unfortunately, Home Directory file system is not optimized for handling heavy computations. If the SRA file is particularly large, you can change the default download path for SRA data to our scratch file system using one of the following two approaches. The following approaches use the /fs/scratch/PAS1234/johndoe/ncbi directory as an example.

Change the prefetch directory using vdb-config

module load sratoolkit/3.0.2
vdb-config -s /repository/user/main/public/root=/fs/scratch/PAS1234/johndoe/ncbi
prefetch SRR390728
srapath SRR390728

You should find the SRR390728 accession at /fs/scratch/PAS1234/johndoe/ncbi/sra/SRR390728.sra

Download to the current directory (available for version 2.10 or later)

module load sratoolkit/3.0.2
vdb-config --prefetch-to-cwd
mkdir -p /fs/scratch/PAS1234/johndoe/ncbi
cd /fs/scratch/PAS1234/johndoe/ncbi
prefetch SRR390728
srapath SRR390728

You should find the SRR390728 accession at /fs/scratch/PAS1234/johndoe/ncbi/SRR390728/SRR390728.sra

Known Issues

Error when downloading SRA data

NCBI now utilizes cloud-style object stores. To access SRA cloud data, please use version 2.10 or later and provide your AWS or GCP access credentials to vdb-config. For more information, please visit https://github.com/ncbi/sra-tools/wiki/04.-Cloud-Credentials. However, you can continue to use older versions to process SRA local data.

 

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

STAR

STAR: Spliced Transcripts Alignment to a Reference.

Availability and Restrictions

Versions

The following versions of STAR are available on OSC clusters:

Version Pitzer Ascend Cardinal
2.7.10b X X X
2.7.11b     X*
* Current default version

You can use module spider star to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

STAR is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Alexander Dobin, Open source

Usage

Usage on Cardinal

Set-up

To configure your environment for use of STAR, run the following command:  module load star/version. For example, use module load star/2.7.10b to load STAR 2.7.10b.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

STAR-CCM+

STAR-CCM+ provides the world’s most comprehensive engineering physics simulation inside a single integrated package. Much more than a CFD code, STAR‑CCM+ provides an engineering process for solving problems involving flow (of fluids and solids), heat transfer and stress. STAR‑CCM+ is unrivalled in its ability to tackle problems involving multi‑physics and complex geometries.  Support is provided by CD-adapco. CD-adapco usually releases new version of STAR-CCM+ every four months.

Availability and Restrictions

Versions

STAR-CCM+ is available on the Cardinal Cluster. The versions currently available at OSC are:

Version Pitzer Cardinal
18.06.006   X
18.06.006-hbm   X
18.06.006-mixed   X*
18.06.006-mixed-hbm   X
19.04.009   X
19.04.009-hbm   X
19.04.009-mixed   X
19.04.009-mixed-hbm   X
19.06.009 X X
19.06.009-hbm   X
19.06.009-mixed X X
19.06.009-mixed-hbm   X
* Current default version

We have STAR-CCM+ Academic Pack, which includes STAR-CCM+, STAR-innovate, CAD Exchange, STAR-NX, STAR-CAT5, STAR-Inventor, STAR-ProE, JTOpen Reader, EHP, Admixturs, Vsim, CAT, STAR-ICE, Battery Design Studio, Sattery Simulation Module, SPEED, SPEED/Enabling PC-FEA, SPEED/Optimate, DARS, STAR-CD, STAR-CD/Reactive Flow Models, STAR-CD/Motion, esiece, and pro-STAR.

You can use module spider starccm  to view available modules for a given machine. The default versions are in double precision. Please check with module spider starccm  to see if there is a mixed precision version available. Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

Academic users can use STAR-CCM+ on OSC machines if the user or user's institution has proper STAR-CCM+ license. Currently, users from Ohio State University, University of Cincinnati, University of Akron, and University of Toledo can access the OSC's license.

Use of STAR-CCM+ for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction. 

Currently, OSC has a 80 seat license (ccmpsuite, which allows up to 80 concurrent users), with 4,000 HPC licenses (DOEtoken) for academic users. 

Access for Commercial Users

Contact OSC Help for getting access to STAR-CCM+ if you are a commercial user.

Publisher/Vendor/Repository and License Type

Siemens, Commercial

Usage

Usage on Cardinal

Set-up on Cardinal

We recommend to run STAR-CCM+ on only the compute nodes. Thus, all STAR-CCM+ jobs should be submitted via the batch scheduling system, either as interactive or non-interactive batch jobs. To load the default version of STAR-CCM+ module on Cardinal, use  module load starccm . To select a particular software version, use   module load starccm/version . For example, use  module load starccm/11.02.010  to load STAR-CCM+ version 11.02.010 on Cardinal.

Batch Usage on Cardinal

When you log into cardinal.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your STAR-CCM+ analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used. STAR-CCM+ can be run on OSC clusters in either interactive mode or in non-interactive batch mode.

Interactive Batch Session

Interactive mode is similar to running STAR-CCM+ on a desktop machine in that the graphical user interface (GUI) will be sent from OSC and displayed on the local machine. To run interactive STAR-CCM+, it is suggested to request necessary compute resources from the login node, with X11 forwarding. The intention is that users can run STAR-CCM+ interactively for the purpose of building their model, preparing input file (.sim file), and checking results. Once developed this input file can then be run in no-interactive batch mode. For example, the following line requests one node with 28 cores( -N 1 -n 28 ), for a walltime of one hour ( -t 1:00:00 ), with one STAR-CCM+ base license token ( -L starccm@osc:1 ) on Cardinal:

sinteractive -N 1 -n 28 -t 1:00:00 -L starccm@osc:1

This job will queue until resources become available. Once the job is started, you're automatically logged in on the compute node; and you can launch STAR-CCM+ GUI with the following commands:

module load starccm
starccm+ -mesa
Non-interactive Batch Job (Serial Run using 1 Base Token)

batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

Below is the example batch script ( job.txt ) for a serial run with an input file ( starccm.sim ) on Cardinal:

#!/bin/bash
#SBATCH --job-name=starccm_test
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L starccm@osc:1

cd $TMPDIR  
cp $SLURM_SUBMIT_DIR/starccm.sim .  
module load starccm  
starccm+ -batch starccm.sim >&output.txt  
cp output.txt $SLURM_SUBMIT_DIR

To run this job on OSC batch system, the above script is to be submitted with the command:

sbatch job.txt
Non-interactive Batch Job (Parallel Run using HPC Tokens)

To take advantage of the powerful compute resources at OSC, you may choose to run distributed STAR-CCM+ for large problems. Multiple nodes and cores can be requested to accelerate the solution time. The following shows an example script if you need 2 nodes with 28 cores per node on Cardinal using the inputfile named   starccm.sim   :

#!/bin/bash
#SBATCH --job-name=starccm_test
#SBATCH --time=3:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L starccm@osc:1,starccmpar@osc:55

cp starccm.sim $TMPDIR
cd $TMPDIR
module load starccm 

srun hostname | sort -n > ${SLURM_JOB_ID}.nodelist

starccm+ -np 56 -batch -machinefile ${SLURM_JOB_ID}.nodelist -mpi openmpi starccm.sim >&output.txt 
cp output.txt $SLURM_SUBMIT_DIR

In addition to requesting the STAR-CCM+ base license token ( -L starccm@osc:1 ), you need to request copies of the  starccmpar  license, i.e., HPC tokens ( -L starccm@osc:1,starccmpar@osc:[n] ), where [n] is equal to the number of cores minus 1.

We recommand using openmpi for your parallel jobs. Especially, 17.02.007 version would not work with intelmpi

HBM Variants

Each version of STARC-CCM+ installed on cardinal also has a -hbm variant. If running on the cpu partition of cardinal, using this variant should run faster by utilizing the HBM memory on the nodes. Our tests show up to a 35% improvement by using this version. Running this on the cache or gpu partitions will not show any benefit.

See Also

Supercomputer: 
Service: 

Run STAR-CCM+ to STAR-CCM+ Coupling

This documentation is to discuss how to run STAR-CCM+ to STAR-CCM+ Coupling simulation in batch job at OSC. The following example demonstrates the process of using STAR-CCM+ version 11.02.010 on Cardinal. Depending on the version of STAR-CCM+ and cluster you work on, there mighe be some differences from the example. Feel free to contact OSC Help if you have any questions. 

Prepare Lagging Simulation

  • Launch the STAR-CCM+ GUI following the instructions on this page
  • Load the simulation that lags and prepare the lagging simulation following the STAR-CCM+ User Guide
    • Active a co-simulation model
    • Set "Concurrency mode -> Method" to Lag
    • Other setups
  • Save the lagging simulation and name it for example as lag.sim 

Prepare Leading Simulation

  • Load the simulation that leads and prepare the leading simulation following the STAR-CCM+ User Guide
    • Active a co-simulation model
    • Set "Concurrency mode -> Method" to Lead
  • Go to the "Connect Method" node by selecting "Co-Simulations -> <name of co-simulation> -> Conditions". Click "Edit" of "Connect Method". In "Connect Method" node, select "Launch Application and Connect" under method. Under "Launch Application and Connect", put the following information as "Launch Command":

 /usr/local/starccm/11.02.010/STAR-CCM+11.02.010-R8/star/bin/starccm+ -load -server -rsh /usr/local/bin/pbsrsh lag.sim

         See the picture below:

connect method

  • Save the leading simulation and name it for example as lead.sim

Prepare Job Script

In the job script, use the following command to run the co-simulation:

starccm+ -np N,M -rsh /usr/local/bin/pbsrsh -batch -machinefile $PBS_NODEFILE lead.sim 

where N is # of cores for the leading simulation and M is # of cores for the lagging simulation, and the summation of N and M should be the total number of cores you request in the job.

Once the job is completed, the output results of the leading simulation will be returned, while the lagging simulation runs on the background server and the final results won't be saved. 

 

 

Supercomputer: 
Service: 

Schrodinger

The Schrodinger molecular modeling software suite includes a number of popular programs focused on drug design and materials science but of general applicability, for example Glide, Jaguar, and MacroModel.  Maestro is the graphical user interface for the suite.  It allows the user to construct and graphically manipulate both simple and complex chemical structures, to apply molecular mechanics and dynamics techniques to evaluate the energies and geometries of molecules in vacuo or in solution, and to display and examine graphically the results of the modeling calculations.

Availability and Restrictions

Versions

The Schrodinger suite is available on Cardinal. The versions currently available at OSC are:

Version Cardinal
2023.2 X
2024.3 X
* Current default version

You can use  module spider schrodinger to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Schrodinger is available to all academic users.  

To use Schrodinger you will have to be added to the license server first.  Please contact OSC Help to be added. Please note that if you are a non-OSU user, we need to send your name, contact email, and affiliation information to Schrodinger in order to grant access. Currently, we have license for following features:

CANVAS_ELEMENTS
CANVAS_MAIN
CANVAS_SHARED
COMBIGLIDE_MAIN
EPIK_MAIN
FFLD_OPLS_MAIN
GLIDE_MAIN
GLIDE_XP_DESC
IMPACT_MAIN
KNIME_MAIN
LIGPREP_MAIN
MAESTRO_MAIN
MMLIBS
MMOD_CONFGEN
MMOD_MACROMODEL
MMOD_MBAE
QIKPROP_MAIN

You need to use one of following software flags in order to use the particular feature of the software without license errors.

macromodel(10), glide(20)[16], ligprep(10), qikprop(10), epik(10)

*The number within the parentheses refers to the total number of licenses for each software flag

*The number within the brackets refers to the number of licenses per group for each software flag

You can add #SBATCH -L glide@osc:1 to your job script if you use GLIDE for example. When you use this software flag, your job won't start until it secures available licenses as there are a limited amount of total licenses and licenses per group. Please read the batch script examples below.  You can check your license usage via the license usage checking tool

Note that OSC has purchased and installed Schrödinger with paid licenses. This doesn't include the Desmond license. We have installed Desmond separately using free licenses.  For more details see our Desmond page.

Publisher/Vendor/Repository and License Type

Schrodinger, LLC/ Commercial

Usage

Usage on Cardinal

To set up your environment for schrodinger load one of its modulefiles:

module load schrodinger/2024.3

Using schrodinger interactively requires an X11 connection. Typically one will launch the graphical user interface maestro.  This can be done natively via the simple command maestroor additionally with either software rendering:

maestro -SGL

or with hardware rendering:

module load virtualgl
vglrun maestro

Note that hardware rendering requires a node with a GPU as well as the additional vglrun syntax above.  In principle hardware rendering is superior; however, in practice it can be laggier, and thus software rendering can yield a better experience.

Here is an example batch script that uses schrodinger non-interactively via the batch system:

#!/bin/bash
# Example glide single node batch script.
#SBATCH --job-name=glidebatch
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=28
#SBATCH -L glide@osc:1

module load schrodinger
cp * $TMPDIR
cd $TMPDIR
host=`srun hostname|head -1`
nproc=`srun hostname|wc -l`
glide -WAIT -HOST ${host}:${nproc} -NJOBS 40 receptor_glide.in
ls -l
cp * $SLURM_SUBMIT_DIR

The glide command passes control to the Schrodinger Job Control utility which processes the two options: The WAIT option forces the glide command to wait until all tasks of the command are completed. This is necessary for the batch jobs to run effectively. The HOST option specifies how tasks are distributed over processors.  In addition, the glide option NJOBS distributes the job into subjobs which can number more than the licenses or processors specified in the batch directives.

Determining the optimal amount of resources will probably require benchmarking.  See the Schrodinger Knowledge Base for advice, e.g., running glide in parallel and docking a large database.  Note also that OSC imposes a usage limit of 16 concurrent glide licenses per group. So while using --ntasks-per-node to request a whole Cardinal node may have significant performance benefits even if all processors are not used, it is not possible to have that many glide licenses.

Known Issues

Maestro 2023.2 will not launch in a Cardinal Desktop

Name: Maestro 2023.2 Desktop
Resolution: Resolved (workaround)
Update: 1/8/2025
Version: 2023.2

Maestro from module schrodinger/2023.2 does not launch in an OnDemand Cardinal Desktop.

Three workarounds are known.   Use module schrodinger/2024.3 in a Cardinal Desktop:

module load schrodinger/2024.3
maestro

Or use hardware rendering in an OnDemand Interactive Apps Lightweight Desktop.  Or select the Schrodinger GUI in the Interactive Apps menu in OnDemand which works with both 2023.2 and 2024.3.

Maestro on Cardinal can have a long pause

Name: Maestro GUI
Resolution: None
Update: 4/24/2025
Version: 2023.2 and 2024.3

Maestro from modules schrodinger/2023.2 and schrodinger/2024.3 presents an 80 second delay after importing complicated ring structures. The delay manifests itself as an unresponsive GUI, but normal function returns.  The delay is reproducible, happens for all versions, happens for all modes of launching (native, software rendering, or hardware rendering), and is independent of the method and location of a user's connection to OSC.

No workarounds are known.

Further Reading

Supercomputer: 
Service: 

Scipion

SCIPION is an image processing framework fo robtaining 3D models of macromolecular complexes using Electron Microscopy (3DEM). It integrates several software packages and presents a unified interface for both biologists and developers. Scipion allows you to execute workflows combining different software tools, while taking care of formats and conversions. Additionally, all steps are tracked and can be reproduced later on.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

Version Ascend
3.7.1 X
* Current default version

You can use module spider scipion to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Scipion is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

All scipion code and plugins, are licensed under the GPL3 (http://www.gnu.org/licenses/gpl-3.0.html)

Now, Scipion interacts, and in some cases installs 3rd party software with its own LICENCE that must be observed.

So, it is under the user responsibility to check the license of each of the software scipion is installing.

In most cases, if not all, software is free available for academic use and industry but there are few exceptions where industry users are not granted for a free usage. You must check each case.

Usage

Usage on Pitzer

Set-up

To configure your environment for use of scipion, run the following command:  module load scipion/version. For example, use module load scipion/3.7.1 to load SCIPION 3.7.1

Plugins

The following plugins are installed

scipion-em-xmipp
scipion-em-resmap
scipion-em-sphire
scipion-em-localrec
scipion-em-bsoft
scipion-em-ccp4
scipion-em-cryoef
scipion-em-spider
scipion-em-imagic

 

Further Reading

Supercomputer: 
Service: 

SnpEff

SnpEff is a variant annotation and effect prediction tool. It annotates and predicts the effects of variants on genes (such as amino acid changes).

Availability and Restrictions

Versions

The following versions of SnpEff are available on OSC clusters:

Version Ascend Cardinal
5.2c X X*
* Current default version

You can use  module spider snpeff to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

SnpEff is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

http://snpeff.sourceforge.net, Open source

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Spark

Apache Spark is an open source cluster-computing framework originally developed in the AMPLab at University of California, Berkeley but was later donated to the Apache Software Foundation where it remains today. In contrast to Hadoop's disk-based analytics paradigm, Spark has multi-stage in-memory analytics. Spark can run programs up-to 100x faster than Hadoop’s MapReduce in memory or 10x faster on disk. Spark support applications written in python, java, scala and R.

Availability and Restrictions

Versions

The following versions of Spark are available on OSC systems: 

Version Pitzer Ascend Cardinal Note
3.5.1 X X X*  
* Current default version

You can use module spider spark to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Spark is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

The Apache Software Foundation, Open source

Usage

Run a Spark Application Using a Job Script

Setting up a Spark cluster

Before running any Spark application, you need to initialize a Spark cluster based on the resources allocated. For example, assume you have allocated two CPU nodes either in an interactive session or through a batch job:

Requesting an Interactive session
salloc -N 2 --exclusive -A <project-code>
A Batch job script
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --exclusive

Once your resources are allocated, you can use the slurm-spark-submit script to set up the Spark cluster:

module load spark/3.5.1
slurm-spark-submit

You should see output similar to the following:

/apps/spack/0.21/ascend/linux-rhel9-zen2/spark/gcc/11.4.1/3.5.1-lbffccn/sbin/start-master.sh
SPARK_MASTER_HOST=a0114.ten.osc.edu
SPARK_MASTER_PORT=7077

And you should see the following line repeated twice in the output:

25/05/14 12:04:29 INFO Worker: Successfully registered with master spark://a0114.ten.osc.edu:7077

This setup starts a Spark master on one of the CPU nodes and launches one Spark worker per node, resulting in a total of two workers. In this configuration, each worker is allocated all the available CPUs and memory on its respective node.

If you want multiple workers per node, you can use the -W option with slurm-spark-submit. For example:

slurm-spark-submit -W 4 -w <number_cpus_per_worker> -m <memory_per_worker>

This command launches four workers per node, resulting in a total of eight workers across two nodes.

Please note that by default, the slurm-spark-submit command allocates all available CPUs and memory on a node to each worker. To prevent overallocation and ensure proper resource distribution, you must explicitly specify the number of CPUs (-w) and memory (-m) for each worker.

Accessing the Spark Web UI

You can monitor the status and resource usage of your Spark cluster through the Spark Web UI. Follow these steps:

  1. Launch a lightweight desktop:
    https://ondemand.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/vdi/session_contexts/new
  2. Once the desktop session starts, open a web browser and enter the value of SPARK_MASTER_HOST obtained from your job output, followed by port 8080. For example: a0114.ten.osc.edu:8080
  3. You should now be connected to the Spark Web UI, where you can view the Spark cluster status, running jobs, and resource consumption.

Running a Spark applicaiton

Once the Spark cluster is ready, you can run a Spark application using slurm-spark-submit, specifying Spark properites for the Spark session:

slurm-spark-submit --no-init \
  --driver-memory 2G \
  --executor-memory 60G \
  --executor-cores 24 \
  /users/PZS0645/support/share/tests/spark/spark_parallel_example.py
Explanation of options
  • --no-init: Do not start a new Spark cluster. Omit this option if you have not set up a Spark cluster as instructed above — in that case, the script will initialize one for you.
  • --driver-memory 2G: Allocates 2 GB of memory for the driver process.
  • --executor-memory 60G: Allocates 60 GB of memory for each executor process.
  • --executor-cores 24: Assigns 24 CPU cores per executor. If each worker node has 96 CPU cores, this configuration allows four executors to run on each worker.

Creating a Spark session in Python

In your Python application, create a Spark session to communicate with the Spark cluster:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
        .appName("MySparkApp") \
        .getOrCreate()

If you run into issues with an uncallable "JavaPackage" object, you may need to install a package called findspark:

pip install findspark
Then add to the top of the python file:
import findspark
findspark.init()

You can now use this spark session to create DataFrames, run SQL queries, and read/write data. For example:

# Create a DataFrame from a JSON file
df = spark.read.json("data/input.json")

# Run a SQL query
df.createOrReplaceTempView("my_table")
result = spark.sql("SELECT * FROM my_table WHERE value > 100")

# Write the result to a CSV file
result.write.csv("data/output.csv")

Configuring the Spark session in a Python application

While creating a Spark session, you can also specify additional Spark properties. For example:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
       .appName("MySparkApp") \
       .config("spark.executor.memory", "120G") \
       .config("spark.executor.cores", "24") \
       .getOrCreate()

Note that Spark properites set in the code can override those passed through the slurm-spark-submit script. For instance, in the example above, each executor will be allocated 120 GB of memory (as specified in the code), potentially overriding a different value (e.g., 60 GB) provided through the slurm-spark-submit script.

Run a Spark Application in a Jupyter Notebook

Launching a Jupyter + Spark app on OSC OnDemand

On OSC OnDemand, you can use the Jupyter + Spark app to easily set up a Spark cluster and run a Spark application within a notebook. For detailed instructions on how to launch Jupyter + Spark using the OSC OnDemand web interface, please visit:

https://www.osc.edu/content/launching_jupyter_spark_app

Choosing a kernel

In a Jupyter + Spark instance, you can choose the default PySpark kernel or use a custom kernel created from your Conda environment. To create a custom kernel, please refer to this guide for details.

Please note that there are some issues with both types of kernels. See Known Issues for more details.

Custom Spark properties

When launching a Jupyter + Spark app, a Spark cluster is automatically configured based on the number of nodes and workers you specify. To enable running a Spark application within a notebook, the PYSPARK_SUBMIT_ARGS environment variable is pre-defined. This variable allows communication between your notebook and the Spark cluster.

Spark Property Default Value
spark.driver.memory The job avaible memory if the driver is launched only on the master node; otherwise, 2 GB
spark.executor.memory 90% of the job available memory divided by the number of workers per node
spark.driver.maxResultSize 0 (unlimited)

There are some Spark properties that are better defined when launching a Spark cluster. Once a Spark cluster is running, these properties are difficult or impossible to change from within a notebook, because updates will not take effect on the worker nodes. These properties include:

  • Driver resources: spark.driver.memory, spark.driver.extraJavaOptions
  • Dependencies: spark.jars, spark.files, spark.pyFiles

To modify or define these properties, you can provide a path to a custom properties file when launching the Jupyter + Spark app. This file will override Spark’s default configuration settings. Below is an example of a custom spark-defaults.conf file:

spark.driver.memory 32G
spark.jars /path/to/spark-nlp-jars/spark-nlp-assembly-6.0.3.jar,/path/to/spark-nlp-jars/spark-nlp-jsl-6.0.3.jar
spark.executor.memory 120G

For executor resources, you can also override the defaults when creating a Spark session in your notebook, for example:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
       .appName("MySparkApp") \
       .config("spark.executor.memory", "120G") \ 
       .config("spark.executor.cores", "24") \ 
       .getOrCreate()

This approach allows for greater customization and performance optimization based on your application’s specific requirements. However, before using custom configurations, ensure the cluster has sufficient resources to accommodate them.

Verifying Spark configuration

To view the active Spark configuration (including default or overridden values), you can run the following command in your notebook:

spark.sparkContext.getConf().getAll()

This returns a list of all active Spark configuration settings, which can help with debugging or performance tuning.

Known Issues

Further Reading

See Also

Supercomputer: 
Service: 
Fields of Science: 

Stata

Stata is a complete, integrated statistical package that provides everything needed for data analysis, data management, and graphics. 32-processor MP version is currently available at OSC.

Availability and Restrictions

Versions

The following versions of Stata are available on OSC systems:

Version Cardinal
18 X
* Current default version

You can use module spider stata to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Only academic OSC users can use the software. OSC has the license for 5 seats concurrently. Each user can use up to 32 cores. In order to access the software, please contact OSC Help to get validated.

Publisher/Vendor/Repository and License Type

StataCorp, LLC, Commercial

Usage

Set-up

To configure your environment on Oakley for the usage of Stata, run the following command:

module load stata

Using Stata

Due to licensing restrictions, Stata may ONLY be used via the batch system on Cardinal. See below for information on how this is done.

Batch Usage

OSC has a 5-user license. However, there is no enforcement mechanism built into Stata. In order for us to stay within the 5-user limit, we require you to run in the context of Slurm and to include this option when starting your batch job (the Slurm system will enforce the 5 user limit):

#SBATCH -L stata@osc:1

Non-Interactive batch example

Use the script below as a template for your usage.

#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH --nodes=1 --ntask-per-node=28
#SBATCH -L stata@osc:1
#SBATCH --job-name=stata

module load stata

stata-mp -b do bigjob

 

 

Further Reading

See Also

Supercomputer: 
Service: 

Subread

The Subread package comprises a suite of software programs for processing next-gen sequencing read data like Subread, Subjunc, featureCounts, and exactSNP.

Availability and Restrictions

Versions

The following versions of Subread are available on OSC clusters:

Version Cardinal
2.0.8 X
* Current default version

You can use  module spider subread to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Subread is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

http://subread.sourceforge.net, Open source

Usage

Usage on Cardinal

Set-up

To configure your environment for use of Subread, run the following command: module load subread. The default version will be loaded. To select a particular Subread version, use module load subread/version. For example, use module load subread/2.0.8 to load Subread 2.0.8.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Texlive

TeX Live is a straightforward way to get up and running with the TeX document production system. It provides a comprehensive TeX system with binaries for most flavors of Unix, including GNU/Linux, macOS, and also Windows. It includes all the major TeX-related programs, macro pacakges, and fonts that are free software, including support for many languages around the world.

Availability and Restrictions

Versions

The following versions are available on OSC clusters:

Version Pitzer Ascend Cardinal
2024   X X*
2025 X    
* Current default version

You can use module spider texlive to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Texlive is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Per the TeX Live licensing, copying, and redistribution webpage, all the material in TeX Live may be freely used, copied, modified, and/or redistributed, subject to (in many cases) the sources remaining freely available.

Please visit this link for full licensing/copyright information.

Usage

Usage

Set-up

To configure your environment for use of mriqc, run the following command:  module load texlive/version. For example, use module load texlive/2021 to load Texlive 2021.

Further Reading

Tag: 
Supercomputer: 
Service: 
Technologies: 
Fields of Science: 

Tinker

Tinker is a molecular modeling package. Tinker provides a general set of tools for molecular mechanics and molecular dynamics.

Availability and Restrictions

Versions

Tinker is available on Cardinal and Pitzer. The versions currently installed at OSC are

Version Cardinal
8.10.5  
8.11.3 X
* Current default version

You can use module spider tinker to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Tinker is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Tinker Core Development Consortium

Usage

Usage on Cardinal

Set-up

To configure your environment for use of Tinker, you may first need to load the correct compiler. Use module spider tinker to see the compatable compilers. Then load a compatable compiler by runningmodule load compiler/version.

Then use the command module load tinker. This will load the default version of Tinker. To select a particular version, use module load tinker/version .

 

For example, execute module load intel/2021.10.0 then module load tinker/8.11.3 to load Tinker version 8.11.3 on Cardinal.

 

Further Reading

Supercomputer: 

Trimmomatic

Trimmomatic performs a variety of useful trimming tasks for illumina paired-end and single ended data.The selection of trimming steps and their associated parameters are supplied on the command line.

Availability and Restrictions

Versions

The following versions of Trimmomatic are available on OSC clusters:

Version Ascend
0.38 X
* Current default version

You can use  module spider trimmomatic to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Trimmomatic is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

http://www.usadellab.org/cms/?page=trimmomatic, Open source

Usage

Usage on Ascend

Set-up

To configure your environment for use of Trimmomatic, run the following command: module load trimmomatic/version. For example, use module load trimmomatic/0.38 to load Trimmomatic 0.38.

Usage

This software provides a wrapper script around a Java executable .jar file. To see the usage, use the following command: trimmomatic. The script uses the variables  $JAVA_ARGS and $JAVA_OPTS. You can set these as environment variables to modify the java arguments, either before invoking the trimmomatic script or on the same line: JAVA_ARGS="some_args" trimmomatic

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

Trinity

Trinity represents a novel method for the efficient and robust de novo reconstruction of transcriptomes from RNA-seq data.

Availability and Restrictions

The following versions of Trinity are available on OSC clusters:

Version Cardinal
2.15.2 X

You can use  module spider trinityrnaseq to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Trinity is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Broad Institute and the Hebrew University of Jerusalem, Open source

Usage

Usage on Cardinal

Set-up

To configure your environment for use of Trinity, run the following command: module load trinityrnaseq/version. For example, use module load trinityrnaseq/2.15.2 to load Trinity 2.15.2.

Usage

Trinity is installed in an Apptainer container. The TRINITY_IMG environment variable contains the container image file path. If you would like to run other Trinity or other Trinity-supported analyses, you can run them by refferring to their path in the container. For example, with TrinityStats.pl:
 
apptainer exec -e $TRINITY_IMG /usr/local/bin/util/TrinityStats.pl 
For convenience, a wrapper script has been provided for the main Trinity program. It can be run as follows:
 
Trinity --version

For more information about Apptainer/Singularity usages, please read the OSC Apptainer/Singularity page.

Further Reading

 
Supercomputer: 
Service: 
Fields of Science: 

TurboVNC

TurboVNC is an implementation of VNC optimized for 3D graphics rendering.  Like other VNC software, TurboVNC can be used to create a virtual desktop on a remote machine, which can be useful for visualizing CPU-intensive graphics produced remotely.

Availability and Restrictions

Versions

The versions currently available at OSC are:

Version Pitzer Ascend Cardinal Notes
3.1.1 X X X*  
* Current default version

You can use  module spider turbovnc to view available modules for a given cluster. Feel free to contact OSC Help  if you need other versions for your work.

Access

TurboVNC is available for use by all OSC users.

Publisher/Vendor/Repository and License Type

https://www.turbovnc.org, Open source

Usage

Usage on Pitzer

Setup on Pitzer

To load the default version of TurboVNC module, use module load turbovnc/version. For example, use module load turbovnc/3.1.1to use TurboVNC 3.1.1. 

Please do not SSH directly to compute nodes and start VNC sessions! This will negatively impact other users (even if you have been assigned a node via the batch scheduler), and we will consider repeated occurances an abuse of the resources. If you need to use VNC on a compute node, please see our HOWTO for instructions.

Using TurboVNC

To start a VNC server on your current host, use the following command:

vncserver  

After starting the VNC server you should see output similar to the following:  

New 'X' desktop is hostname:display
Starting applications specified in /nfs/nn/yourusername/.vnc/xstartup.turbovnc
Log file is /nfs/nn/yourusername/.vnc/hotsname:display.log

Make a note of the hostname and display number ("hostname:display"), because you will need this information later in order to connect to the running VNC server.  

To establish a standard unencrypted connection to an already running VNC server, X11 forwarding must first be enabled in your SSH connection.  This can usually either be done by changing the preferences or settings in your SSH client software application, or by using the -X or -Y option on your ssh command.     

Once you are certain that X11 forwarding is enabled, create your VNC desktop using the vncviewer command in a new shell.

vncviewer

You will be prompted by a dialogue box asking for the VNC server you wish to connect to.  Enter "hostname:display".  

You may then be prompted for your HPC password.  Once the password has been entered your VNC desktop should appear, where you should see all of your home directory contents. 

When you are finished with your work on the VNC desktop, you should make sure to close the desktop and kill the VNC server that was originally started.  The VNC server can be killed using the following command in the shell where the VNC server was originally started:

vncserver -kill :[display]

For a full explanation of each of the previous commands, type man vncserver or man vncviewer at the command line to view the online manual.

Further Reading

Additional information about TurboVNC can be found at the VirtualGL Project's documentation page.  

See Also

Supercomputer: 
Service: 
Fields of Science: 

VASP

The Vienna Ab initio Simulation Package, VASP, is a suite for quantum-mechanical molecular dynamics (MD) simulations and electronic structure calculations.

Availability and Restrictions

Access

Due to licensing considerations, OSC does not provide general access to this software.

However, we are available to assist with the configuration of individual research-group installations on all our clusters. See the VASP FAQ page for information regarding licensing.

Usage

Using VASP

See the VASP documentation page for tutorial and workshop materials.

Building and Running VASP

If you have a VASP license you may build and run VASP on any OSC cluster. The instructions given here are for VASP 5.4.1; newer 5 versions should be similar; and we have several reports that these worked for VASP 6.3.2 and one report using IntelMPI for VASP 6.5.1.

Most VASP users at OSC run VASP with MPI and without multithreading. If you need assistance with a different configuration, please contact oschelp@osc.edu.  Note that we recommend submitting a batch job for testing because running parallel applications from a login node can be problematic.

You can build and run VASP using either IntelMPI or MVAPICH. Performance is similar for the two MPI families. Instructions are given for both. The IntelMPI build is simpler and more standard. MVAPICH is the default MPI installation at OSC; however, VASP had failures with some prior MVAPICH2 versions, so building with the newest MVAPICH, in particular 3.0 or newer, is recommended.

Build instructions assume that you have already unpacked the VASP distribution and patched it if necessary and are working in the vasp directory. It also assumes that you have the default module environment loaded at the start.

Building with IntelMPI

1. Copy arch/makefile.include.linux_intel (for VASP 6.5 as of August 2025 use arch/makefile.include.oneapi) and rename it makefile.include.

2. Edit makefile.include to replace the two lines

OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o \
$(MKLROOT)/interfaces/fftw3xf/libfftw3xf_intel.a

with one line

OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

3. Make sure the FCL line is

FCL = mpiifort -mkl=sequential

4. Load modules and build the code (using the latest IntelMPI may yield the best performance; for VASP 5.4.1 the modules were intel/19.0.5 and intelmpi/2019.3 as of October 2019, but those modules no longer exist)

module load intel-oneapi-mpi/2021.10.0
make

5. Add the modules used for the build, e.g., module load intel-oneapi-mpi/2021.10.0, to your job script.

Building with MVAPICH

1. Copy arch/makefile.include.linux_intel and rename it makefile.include.

2. Edit makefile.include to replace mpiifort with mpif90

FC         = mpif90
FCL        = mpif90 -mkl=sequential

3. Replace the BLACS, SCALAPACK, OBJECTS, INCS and LLIBS lines with

BLACS      =
SCALAPACK  = $(SCALAPACK_LIBS)

OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
INCS       = $(FFTW3_FFLAGS)

LLIBS      = $(SCALAPACK) $(FFTW3_LIBS_MPI) $(LAPACK) $(BLAS)

4. Load modules and build the code (using the latest MVAPICH is recommended; for VASP 5.4.1 the modules were intel/19.0.5 and mvapich2/2.3.2 as of October 2019)

module load scalapack
module load fftw3
make

5. Add the modules used for the build, e.g., module load scalapack fftw3, to your job script.

Building for GPUs

The "GPU Stuff" section in arch/makefile.include.linux_intel_cuda is generic.  It can be updated for OSC clusters using the environment variables defined by a cuda module.  The OSC_CUDA_ARCH environment variables defined by cuda modules on all clusters show the specific CUDA compute capabilities.  Below we have combined them as of February 2023 so that the resulting executable will run on any OSC cluster.  In addition to the instructions above, here are the specific CUDA changes and the commands for building a gpu executable.

Edits:

CUDA_ROOT         = $(CUDA_HOME)
GENCODE_ARCH      = -gencode=arch=compute_70,code=\"sm_70,compute_70\" \
                    -gencode=arch=compute_80,code=\"sm_80,compute_80\" \
                    -gencode=arch=compute_90,code=\"sm_90,compute_90\"

Commands:

module load cuda
make gpu

See this VASP 5.4.1 Manual page for details through versions 6.2, this VASP 6.2.0 Manual page for details of the newer OpenACC GPU port, and this NVIDIA page for reference.

Running VASP generally

Be sure to load the appropriate modules in your job script based on your build configuration, as indicated above. If you have built with -mkl=sequential you should be able to run VASP as follows:

mpiexec path_to_vasp/vasp_std

If you have a problem with too many threads you may need to add this line (or equivalent) near the top of your script:

export OMP_NUM_THREADS=1

Running VASP with GPUs

See this VASP 5.4.1 Manual pagethis VASP 6.2.0 Manual page  and this VASP Scaling NVIDIA page for feature restrictions, input requirements, and performance tuning examples.  To acheive maximum performance, benchmarking of your particular calculation is essential.  As a point of reference, although GPUs are the scarce resource, some user reports are that optimal performance is with 3 or 4 MPI ranks per GPU.  This is expected to depend on method and simulation size.

If you encounter a CUDA error running a GPU enabled executable, such as:

CUDA Error in cuda_mem.cu, line 44: all CUDA-capable devices are busy or unavailableFailed to register pinned memory!

then you may need to use the default compute mode which can be done by adding this line (or equivalent) near the top of your script, e.g., for Cardinal:

#SBATCH --nodes=1 --ntasks-per-node=48 --gpus-per-node=1 --gpu_cmode=shared

 

Known Issues

None

There are presently no known issues.
 

Further Reading

See Also

Service: 

VCFtools

VCFtools is a program package designed for working with VCF files, such as those generated by the 1000 Genomes Project. The aim of VCFtools is to provide easily accessible methods for working with complex genetic variation data in the form of VCF files.

Availability and Restrictions

The following versions of VCFtools are available on OSC clusters:

Version Pitzer Ascend Cardinal
0.1.16 X X X*
* Current default version

You can use  module spider vcftools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

VCFtools is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Adam Auton, Petr Danecek, Anthony Marcketta/ Open source

Usage

Usage on Pitzer

Set-up

To configure your environment for use of VCFtools, run the following command: module load vcftools/version. For example, use module load vcftools/0.1.16 to load VCFtools 0.1.16.

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

VMD

VMD is a visulaization program for the display and analysis of molecular systems.

Availability and Restrictions

Versions

The following versions of VMD are available on OSC clusters:

Version Cardinal
1.9.3 X
1.9.4a55 X*
* Current default version

Access

VMD is for academic purposes only. Please review the license agreement before you use this software.

Publisher/Vendor/Repository and License Type

TCBG, Beckman Institute/ Open source

Usage

Usage on Cardinal

Using VMD with OSC OnDemand

It is recommended to use VMD with OSC OnDemand. On the OnDemand page launch the VMD GUI from the interactive apps dropdown menu. This will open the VMD Main, OpenGL Display, and terminal windows. End a session through the VMD Main window by selecting File → Quit.

See VMD Tutorials for basic VMD usage instructions.

Further Reading 

Supercomputer: 
Technologies: 
Fields of Science: 

VirtualGL

VirtualGL allows OpenGL applications to run with 3D hardware accerlation.

Availability & Restrictions

Versions

The following versions of VirtualGL are available on OSC clusters:

Version Pitzer Ascend Cardinal Notes
3.1.1 X X X*  
* Current default version

Access

OSC provides VirtualGL to all OSC users.

Publisher/Vendor/Repository and License Type

Julian Smart, Robert Roebling et al., Open source, LGPL v2.1

Usage

Usage

Set-up

Configure your environment for use of VirtualGL with  module load virtualgl/version. For example, use module load virtualgl/3.1.1 to load VirtualGL 3.1.1.

Run a OpenGL program

User must invoke vglrun command to run a OpenGL program with VirtualGL in a Virtual Desktop Interface (VDI) app or an Interactive HPC 'vis' type Desktop app, e.g.

$ module load virtualgl
$ vglrun glxinfo |grep OpenGL
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: Tesla V100-PCIE-16GB/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 450.80.02
OpenGL core profile shading language version string: 4.60 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6.0 NVIDIA 450.80.02
OpenGL shading language version string: 4.60 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 450.80.02
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

Further Reading 

Supercomputer: 

VisIt

VisIt is an Open Source, interactive, scalable, visualization, animation and analysis tool for visualizing data defined on two- and three-dimensional structured and unstructured meshes.

 

Availability and Restrictions

Versions

The following versions of VisIt are available on OSC systems: 

Version Pitzer Ascend Cardinal
3.3.3   X X*
3.4.2 X X  
* Current default version

You can use module spider visit to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

VisIt is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Lawrence Livermore National Laboratory, BSD-3 License

Usage

Set-up

We recomend users to run VisIt locally and connect to OSC clusters for data analysis. In this Client-Server Mode, users can visualize data stored on the clusters without download it. 

Install VisIt locally 

Downloadand install a binary distribution locally. The supported versions on OSC clusters are listed above. If you are using an unmatched version, there might be a compatibility issue. During the installation, you will be asked to pick a host profile from a list of computing centers. Please select Ohio Supercomputer Center (OSC) network to continue. If you are using any version prior to 3.2.2, the existing OSC profile is outdated and is not compatible with the current batch scheduler. Please refer to the following section to obtain the up-to-date profiles.

Update host profiles (for version prior to 3.2.2)

Please download the new OSC profiles for Pitzer and place them in $HOME/.visit/hosts if you are using macOS or Linux, or in <visit_installaion>\hostsAfter relaunching VisIt, you should see new profiles named OSC Pitzer.

Further Reading

Tag: 
Supercomputer: 
Service: 
Fields of Science: 

WARP3D

From WARP3D's webpage:

WARP3D is under continuing development as a research code for the solution of large-scale, 3-Dsolid models subjected to static and dynamic loads. The capabilities of the code focus on 
fatigue & fracture analyses primarily in metals. WARP3D runs on laptops-to-supercomputers and can analyze models with several million nodes and elements. 

Availability and Restrictions

Versions

The following versions of WARP3D are available on OSC clusters:

Version Cardinal
18.4.0 X
q* Default version depends on the compiler and MPI version loaded

You can use module spider warp3d to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access 

WARP3D is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

University of Illinois at Urbana-Champaign, Open source

Usage

Usage on Cardinal

Setup on Cardinal

To configure the Cardinal cluster for the use of WARP3D, use the following commands:

module load intel
module load intelmpi
module load warp3d

Batch Usage on Cardinal

Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for OakleyQueues and Reservations for Ruby, and Scheduling Policies and Limits for more info.

Running WARP3D

Below is an example batch script (job.txt) for using WARP3D:

#!/bin/bash
#SBATCH --job-name WARP3D 
#SBATCH --nodes=1 --ntasks-per-node=40 
#SBATCH --time=30:00
#SBATCH --account <project-account>

# Load the modules for WARP3D
module load intel
module load intelmpi
module load warp3d
# Copy files to $TMPDIR and move there to execute the program
cp $WARP3D_HOME/example_problems_for_READMEs/mt_cohes_*.inp $TMPDIR
cd $TMPDIR
# Run the solver using 4 MPI tasks and 6 threads per MPI task 
$WARP3D_HOME/warp3d_script_linux_hybrid 4 6 < mt_cohes_4_cpu.inp
# Finally, copy files back to your home directory 
cp -r * $SLURM_SUBMIT_DIR

In order to run it via the batch system, submit the job.txt file with the following command:

sbatch job.txt

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

WCStools

WCStools is a program package designed for working with Images and the World Coordinate System. The aim of WCStools is to provide methods for relating pixels taken common astronomical images to sky coordinates.

Availability and Restrictions

WCStools is not currently available on any OSC cluster.

Version
3.9.7
* Current default version

You can use  module spider wcstools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

WCStools is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Jessica Mink, Smithsonian Astrophysical Observatory/ Open source

Usage

Usage on Pitzer

Set-up

To configure your environment for use of WCStools, run the following command: module load wcstools. The default version will be loaded. To select a particular WCStools version, use module load wcstools/version . For example, use module load wcstools/3.9.7 to load WCStools 3.9.7.

Further Reading

Service: 
Fields of Science: 

XFdtd

XFdtd is an electromagnetic simulation solver. Its features analyze problems in antenna design and placement, biomedical and SAR, EMI/EMC, microwave devices, radar and scattering, automotive radar, and more.

Availability and Restrictions

Versions

The following versions of XFdtd are available on OSC clusters:

Version Cardinal
7.10.2.3 X*
7.11.0.3 X
* Current default version

You can use  module spider xfdtdto view available modules for a given machine. We have a perpetual license file for the currently installed versions but without maintenance license. Thus, our support for XFdtd would be limited including version updates. 

Access

Use of xfdtd for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction. 

Publisher/Vendor/Repository and License Type

Remcom Inc., Commercial

Usage

Usage on Cardinal

Set-up

To configure your environment for use of XFdtd, run the following command: module load xfdtd. The default version will be loaded. To specify a particular version, use the following command: module load xfdtd/version.

Further Reading

Supercomputer: 

aocc

The AMD Optimizing C/C++ and Fortran Compilers (“AOCC”) are a set of production compilers optimized for software performance when running on AMD host processors using the AMD “Zen” core architecture.  Supported processor families are AMD EPYC™, AMD Ryzen™, and AMD Ryzen™ Threadripper™ processors.  The AOCC compiler environment simplifies and accelerates development and tuning for x86 applications built with C, C++, and Fortran languages.

Availability and Restrictions

Versions

aocc is available on the Pitzer and Ascend Cluster. The versions currently available at OSC are:

Version Ascend
2.3.0  
4.2.0 X
5.0.0 X

 

You can use module spider aocc to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

aocc is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

AMD, Please review the license agreement carefully before use.

Usage

Supercomputer: 

bedtools

Collectively, the bedtools utilities are a swiss-army knife of tools for a wide-range of genomics analysis tasks. The most widely-used tools enable genome arithmetic: that is, set theory on the genome. While each individual tool is designed to do a relatively simple task, quite sophisticated analyses can be conducted by combining multiple bedtools operations on the UNIX command line.

Availability and Restrictions

Versions

The following versions of bedtools are available on OSC clusters:

Version Ascend Cardinal
2.31.0 X X
* Current default version

The bedtools module has been renamed to bedtool2 as of 2.18.0 (13-Dec-2013).

You can use module spider bedtools2 to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

bedtools is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Aaron R. Quinlan and Neil Kindlon, Open source

Usage

Further Reading

Supercomputer: 
Service: 
Fields of Science: 

dcm2nii

dcm2niix is designed to convert neuroimaging data from the DICOM format to the NIfTI format. The DICOM format is the standard image format generated by modern medical imaging devices. However, DICOM is very complicated and has been interpreted differently by different vendors. The NIfTI format is popular with scientists, it is very simple and explicit. However, this simplicity also imposes limitations (e.g. it demands equidistant slices). dcm2niix is also able to generate a BIDS JSON format sidecar which includes relevant information for brain scientists in a vendor agnostic and human readable form. The Neuroimaging DICOM and NIfTI Primer provides details.

Availability and Restrictions

Versions

dcm2nii is available on the Pitzer Cluster. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
02_02_2024     X*
11_12_2024 X X  

* Current default version

You can use module spider dcm2nii to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access: Anyone Can Use

All users can use dcm2nii at OSC. If you have any questions, please contact OSC Help

Publisher/Vendor/Repository and License Type

This software is open source. The bulk of the code is covered by the BSD license. Some units are either public domain (nifti*.*, miniz.c) or use the MIT license (ujpeg.cpp). See the source GitHub repository for more details.

Supercomputer: 

fMRIPrep

fMRIPrep is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting.

Availability and Restrictions

Versions

The following versions of fMRIPrep are available on OSC systems: 

Version Pitzer Ascend Cardinal
20.2.0   X X
24.1.1 X X X*
* Current default version

You can use module spider fMRIPrep to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

fMRIPrep is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Developed at Poldrack Lab at Stanford University, for use at the Center for Reproducible Neuroscience (CRN), as well as for open-source software distribution.

fMRIPrep uses the 3-clause BSD license; the full license may be found in the LICENSE file in the fMRIPrep distirbution.

All trademarks referenced herein are property of their respective holders.

Copyright (c) 2015-2020, the fMRIPrep developers and the CRN. All rights reserved.

Usage

Usage on Pitzer

Set-up

To configure your environment for use of fMRIPrep, run the following command: module load fmriprep. The default version will be loaded. To select a particular fMRIPrep version, use module load fmriprep/version. For example, use module load fmriprep/20.2.0 to load fMRIPrep 20.2.0.

fMRIPrep is installed in a singularity container.  FMRIPREP_IMG environment variable contains the container image file path. So, an example usage would be

module load fmriprep
singularity exec $FMRIPREP_IMG fmriprep --help

For more information about singularity usages, please read OSC singularity page.

Further Reading

 

Supercomputer: 
Service: 
Fields of Science: 

ffmpeg

FFmpeg is a free software project, the product of which is a vast software suite of libraries and programs for handling video, audio, and other multimedia files and streams.

Availability and Restrictions

Versions

The following versions of FFmpeg are available on OSC clusters:

Version Pitzer Ascend Cardinal
4.3.2 X X X
6.1.1    X X*
6.1.2 X    
* Current default version

You can use  module spider ffmpeg to view available modules for a given machine. The static version is built by John Van Sickle, providing full FFmpeg features.  The non-static version is built on OSC systems and is useful for code development.  Feel free to contact OSC Help if you need other versions for your work.

Access for Academic Users

FFmpeg is available to all OSC users.  

Publisher/Vendor/Repository and License Type

https://www.ffmpeg.org/ Open source (academic)

Usage

Usage on Ascend

Set-up

To configure your environment for use of FFmpeg, run the following command:  module load ffmpeg/version. For example, use module load ffmpeg/4.3.2 to load version 4.3.2.
 
Further Reading
Supercomputer: 
Service: 
Fields of Science: 

oneAPI

oneAPI is an open, cross-industry, standards-based, unified, multiarchitecture, multi-vendor programming model that delivers a common developer experience across accelerator architectures – for faster application performance, more productivity, and greater innovation. The oneAPI initiative encourages collaboration on the oneAPI specification and compatible oneAPI implementations across the ecosystem.

Availability and Restrictions

Versions

oneAPI is available on  Pitzer, Ascend, and Cardinal. The versions currently available at OSC are:

Version Pitzer Ascend Cardinal
2023.2.3 X X X
2024.0.2     X
2024.1.0 X X X*
2025.0.4 X X X
* Current Default Version

You can use module spider oneapi to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

oneAPI is available to all OSC users. If you have any questions, please contact OSC Help.

Publisher/Vendor/Repository and License Type

Intel, see Intel's End User License Agreement page for information on the Licensing.

Usage

Tag: 
Supercomputer: 

parallel-command-processor

There are many instances where it is necessary to run the same serial program many times with slightly different input. Parametric runs such as these either end up running in a sequential fashion in a single batch job, or a batch job is submitted for each parameter that is varied (or somewhere in between.) One alternative to this is to allocate a number of nodes/processors to running a large number of serial processes for some period of time. The command parallel-command-processor allows the execution of large number of independent serial processes in parallel. parallel-command-processor works as follows: In a parallel job with N processors allocated, the PCP manager process will read the first N-1 commands in the command stream and distribute them to the other N-1 processors. As processes complete, the PCP manager will read the next one in the stream and send it out to an idle processor core. Once the PCP manager runs out of commands to run, it will wait on the remaining running processes to complete before shutting itself down.

Alternative with SLURM

With SLURM, you can use srun to run multiple programs using the --multi-prog option and the appropriate configuration. You can find an example at MULTIPLE PROGRAM CONFIGURATION.

Availability and Restrictions

Parallel-Command-Processor is available for all OSC users.

Publisher/Vendor/Repository and License Type

Ohio Supercomputer Center, Open source

Usage

Here is an interactive batch session that demonstrates the use of parallel-command-processor with a config file, pconf. pconf contains several lines of simple commands, one per line. The output of the commands were redirected to individual files.

-bash-3.2$ sinteractive -A <project-account> -N 2 -n 8  
-bash-3.2$ cp pconf $TMPDIR
-bash-3.2$ cd $TMPDIR
-bash-3.2$ cat pconf
ls / > 1 
ls $TMPDIR > 2 
ls $HOME > 3 
ls /usr/local/ > 4 
ls /tmp > 5 
ls /usr/src > 6 
ls /usr/local/src > 7
ls /usr/local/etc > 8 
hostname > 9 
uname -a > 10 
df > 11
-bash-3.2$ module load pcp
-bash-3.2$ srun parallel-command-processor pconf
-bash-3.2$ pwd
/tmp/pbstmp.1371894 
-bash-3.2$ srun --ntasks=2 ls -l $TMPDIR 
854 total 16 
-rw------- 1 yzhang G-3040 1082 Feb 18 16:26 11
-rw------- 1 yzhang G-3040 1770 Feb 18 16:26 4 
-rw------- 1 yzhang G-3040 67 Feb 18 16:26 5
-rw------- 1 yzhang G-3040 32 Feb 18 16:26 6 
-rw------- 1 yzhang G-3040 0 Feb 18 16:26 7 
855 total 28
-rw------- 1 yzhang G-3040 199 Feb 18 16:26 1
-rw------- 1 yzhang G-3040 111 Feb 18 16:26 10
-rw------- 1 yzhang G-3040 12 Feb 18 16:26 2
-rw------- 1 yzhang G-3040 87 Feb 18 16:26 3 
-rw------- 1 yzhang G-3040 38 Feb 18 16:26 8
-rw------- 1 yzhang G-3040 20 Feb 18 16:26 9
-rw------- 1 yzhang G-3040 163 Feb 18 16:25 pconf 
-bash-3.2$ exit

As the command "srun --ntasks=2 ls -l $TMPDIR" shows, the output files are distributed on the two nodes. In a batch file, pbsdcp/sgather can be used to distribute-copy the files to $TMPDIR on all nodes of the job and gather output files once execution has completed. This step is important due to the load that executing many processes in parallel can place on the user home directories.

Here is a slightly more complex example showing the usage of parallel-command-processor and pbsdcp/sgather:

#!/bin/bash
#SBATCH  --nodes=13 --ntasks-per-node=4 
#SBATCH --time=1:00:00 
#SBATCH -A <project-account> 


date

module load biosoftw 
module load blast

set -x

pbsdcp -s query/query.fsa.* $TMPDIR 
pbsdcp -s db/rice.* $TMPDIR 
cd $TMPDIR

for i in $(seq 1 49)

do 
      cmd="blastall -p blastn -d rice -i query.fsa.$i -o out.$i" 
      echo ${cmd} >> runblast 
done

module load pcp
srun parallel-command-processor runblast

mkdir $SLURM_SUBMIT_DIR/output 
sgather -r $TMPDIR $SLURM_SUBMIT_DIR/output

date

Further Reading

The parallel-command-processor command is documented as a man page: man parallel-command-processor.

Service: 

vLLM

vLLM is an open-source inference server for large language models (LLMs).

vLLM is in early user testing phase - not all functionality is guaranteed to work.  Contact oschelp@osc.edu with any questions.
vLLM is not currently suitable for use with protected or sensitive data - do not use if you need protected data service. See https://www.osc.edu/resources/protected_data_service for more details.

Availability and Restrictions

Versions

vLLM is available on OSC Clusters. The versions currently available at OSC are:

Version Cardinal Ascend
0.12.0 X X

 

You can use module spider vllm to view available modules for a given machine.

Access:

All OSC users may use vLLM, but individual models may have their own license restrictions.

Publisher/Vendor/Repository and License Type

Apache-2 license: https://github.com/vllm-project/vllm?tab=Apache-2.0-1-ov-file#readme

Prerequisites

  • GPU Usage: vLLM should be run with a GPU for best performance. 

Due to the need for GPUs, we recommend not running vLLM on login nodes nor OnDemand lightweight desktops.

Running vLLM Overview

1. Load module

2. Start vLLM

 

Commands

vLLM is available through the module system and must be loaded prior to running any of the commands below:

loading vllm module:
module load vllm/0.12.0
Starting vllm:
vllm_start <model_name>

Model names follow the HuggingFace format, e.g., "meta-llama/Llama-3.2-3B".

If the model is available and the service starts successfully, this will print out a port number for the vLLM service. 

VLLM_API_PORT: 61234

This port number is only an example - your port number will differ from the one above.

The VLLM_API_PORT environment variable will be used to define the API endpoint.

Stopping vllm:

vllm can be manually stopped with the following commands:

vllm_stop

It is also killed upon module unload.  If you want to stop the services, you can simply unload the vllm module:

module unload vllm

Model Management

By default, vLLM uses a central, read-only model repository defined by VLLM_CACHE_DIR, offering clients the use of a small number of well-performing, curated models.

However, you can use custom models and manage your own set of models by setting VLLM_CACHE_DIR to a path you have write access to, such as a project directory or scratch space.  This must be done prior to starting vLLM.

export VLLM_CACHE_DIR=/fs/project/ABC1234/vllm/models
vllm_start <model_name>
installing a model:

Upon running vllm_start <model_name>, the target model is automatically pulled to the currently defined VLLM_CACHE_DIR location if it does not already exist. 

You cannot use custom models unless you have not redefined your VLLM_CACHE_DIR prior to starting vLLM, as the default model path is read-only. 
Downloading large LLMs can exceed your disk space quota.  Check model sizes before downloading!


Some models require licensing agreements or are otherwise restricted and require a Hugging Face account and login.  With the vLLM module loaded, use the huggingface-cli tool to login:

hf auth login

You will need your Hugging Face token.  For more details, see https://huggingface.co/docs/huggingface_hub/en/guides/cli.

 

Batch Usage

The vLLM module can be used in batch mode by loading the module in your batch script.  For example, you may want to run offline inference by running a script that relies on an inference endpoint.

vLLM provides an OpenAI API-compliant API endpoint, and can be accessed an OpenAI API-compliant client, meaning you can bring your own clients or write your own.  As long as you can send requests to localhost:$VLLM_API_PORT/v1/, this should work and support a wide variety of workflows. 

For the most up-to-date API compatibility information (and more examples), see: vLLM API

vLLM supports a number of portions of the OpenAI API, including Completions, Chat Completions, Embeddings, and more, but does not currently support the complete OpenAI API, including tools and responses.

Here is a basic Python example using the OpenAI package:

import os
from openai import OpenAI

ollama_port = os.getenv("VLLM_API_PORT")

client = OpenAI( base_url = f"http://localhost:{VLLM_API_PORT}/v1", api_key="") 

response = client.chat.completions.create(
    model = "gemma3:12b",
    messages = [
        {"role": "developer", "content": "talk like a pirate"},
        {"role": "user", "content": "how do I check a Python object's type?"}
     ]
)

For more advanced API usage example with asynchronous requests, see this GitHub project: OSC/async_llm_api 

Please note this software is in early user testing and might not function as desired.  Please reach out to oschelp@osc.edu with any issues.

Jupyter Usage

This is under development - contact oschelp@osu.edu if you're interested in this functionality.

Supercomputer: