Client Resources

Getting Started

Getting Started icon

Welcome to OSC! If you are new to supercomputing, new to OSC, or simply interested in getting an account (if you don't already have one), we have some resources to help you.

New User Resource Guide

Welcome to OSC! This document provides some useful links to get you started.

If you are new to supercomputing, or to OSC, please visit http://www.osc.edu/supercomputing/getting-started for information about how our supercomputers are configured, and some guidance about appropriate usage of the various resources.

Logging in to OSC resources is primarily accomplished via SSH (or “secure shell”). This is a method of getting a command line interface on the HPC, which is running a version of the Linux operating system. For more details on how to connect to our systems, please visit

            http://www.osc.edu/supercomputing/getting-started/getting-connected

If that sounds too intimidating, and you would like to try a zero-client web-browser approach to using OSC, you can log in to OnDemand:

            https://ondemand.osc.edu/

SFTP (Secure File Transfer Protocol) is the primary method of moving files to and from OSC storage systems. SCP (Secure CoPy) is another common method. Small files can be moved to and from OSC systems inside of OnDemand. Moving large files to or from OSC via the login nodes can experience difficulty, since there are compute restrictions on those nodes, so the recommendation is to connect to sftp.osc.edu, or scp.osc.edu. If you have a large amount of data to move, you can schedule transfers via Globus Online.

Batch scripts – instructions to be executed by the node(s) assigned to you when your request for resources can be met by the scheduler – can get very complicated, so it’s best to refer to the documentation for the software you want to use to get started.

            http://www.osc.edu/supercomputing/software/

We also have a more detailed guide on the batch system that we recommend new users read. It provides information about scheduling, and how to write scripts to accomplish various tasks.

            http://www.osc.edu/supercomputing/batch-processing-at-osc

Account Maintenance and Usage: Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details. 

Need Help? Support can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800). 
Fill out a request online. 
 

We regulary maintain our webpages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 

Useful Links:

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: http://www.osc.edu/supercomputing/

Documentation Attachment: 

HPC Basics

Overview

Using the HPC or “supercomputer” is a little different from running programs on your desktop. When you login you’ll be connected to one of the system’s “login nodes”. These nodes serve as a staging area for you to marshal your data and submit jobs to the batch scheduler. Your job will then wait in a queue along with other researchers' jobs. Once the resources it requires become available, the batch scheduler will then run your job on a subset of our hundreds of “compute nodes”. You can see the overall structure in the diagram below.

 

Diagram: Several connected parts illustrating the layout of an OSC cluster. Users connect to one of a few "login nodes", which in turn connect to the "batch system", which runs jobs on a subset of the "compute nodes". The "shared filesystem" is connected to both the login nodes and the compute nodes.

HPC Citizenship

An important point about the diagram above is that OSC clusters are a collection of shared, finite resources. When you connect to the login nodes, you are sharing their resources (CPU cycles, memory, disk space, network bandwidth, etc.) with a few dozen other researchers. The same is true of the file servers when you access your home or project directories, and can even be true of the compute nodes.

For most day-to-day activities you should not have to worry about this, and we take precautions to limit the impact that others might have on your experience. That said, there are a few use cases that are worth watching out for:

  • The login nodes should only be used for light computation; any CPU- or memory-intensive operations should be done using the batch system. A good rule of thumb is that if you wouldn't want to run a task on your personal desktop because it would slow down other applications, you shouldn't run it on the login nodes. (See also: Interactive Jobs.)

  • I/O-intensive jobs should copy their files to fast, temporary storage, such as the local storage allocated to jobs or the Scratch parallel filesystem.

  • When running memory-intensive or potentially unstable jobs, we highly recommend requesting whole nodes.  By doing so you prevent other users jobs from being impacted by your job.

  • If you request partial nodes, be sure to consider the amount of memory available per core. (See: HPC Hardware.) If you need more memory, request more cores. It is perfectly acceptable to leave cores idle in this situation; memory is just as valuable a resource as processors.

In general, we just encourage our users to remember that what you do may affect other researchers on the system. If you think something you want to do or try might interfere with the work of others, we highly recommend that you contact us at oschelp@osc.edu.

Getting Connected

There are two ways to connect to our systems. The traditional way will require you to install some software locally on your machine, including an SSH client, SFTP client, and optionally an X Windows server. The alternative is to use our zero-client web portal, OnDemand.

OnDemand Web Portal

OnDemand is our "one stop shop" for access to our High Performance Computing resources. With OnDemand, you can upload and download files, create, edit, submit, and monitor jobs, run GUI applications, and connect via SSH, all via a web broswer, with no client software to install and configure.

You can access OnDemand by pointing a web browser to ondemand.osc.edu. Documentation is available here. Any newer version of a common web brower should be sufficient to connect.

Using Traditional Clients

Required Software

In order to use our systems, you'll need two main pieces of software: an SFTP client and an SSH client.

SFTP ("SSH File Transfer Protocol") clients allow you transfer files between your workstation and our shared filesystem in a secure manner. We recommend the following applications:

  • FileZilla: A high-performance open-source client for Windows, Linux, and OS X. A guide to using FileZilla is available here (external).
  • CyberDuck: A high quality free client for Windows and OS X.
  • sftp: The command-line utility sftp comes pre-installed on OS X and most Linux systems.

SSH ("Secure Shell") clients allow you to open a command-line-based "terminal session" with our clusters. We recommend the following options:

  • PuTTY: A simple, open-source client for Windows.
  • Secure Shell for Google Chrome: A free, HTML5-based SSH client for Google Chrome.
  • ssh: The command-line utility ssh comes pre-installed on OS X and most Linux systems.

A third, optional piece of software you might want to install is an X Windows server, which will be necessary if you want to run graphical, windowed applications like MATLAB. We recommend the following X Windows servers:

  • Xming: Xming offers a free version of their X Windows server for Microsoft Windows systems.
  • X-Win32: StarNet's X-Win32 is a commercial X Windows server for Microsoft Windows systems. They offer a free, thirty-day trial.
  • X11.app/XQuartz: X11.app, an Apple-supported version of the open-source XQuartz project, is freely available for OS X. 

In addition, for Windows users, you can use OSC Connectwhich is a native windows application developed by OSC to provide a launcher for secure file transfer, VNC, terminal, and web based services, as well as preconfigured management of secure tunnel connections. See this page for more information on OSC Connect. 

Connecting via SSH

The primary way you'll interact with the OSC clusters is through the SSH terminal. See our supercomputing environments for the hostnames of our current clusters. You should not need to do anything special beyond entering the hostname.

Once you've established an SSH connection, you will be presented with some informational text about the cluster you've connected to followed by a UNIX command prompt. For a brief discussion of UNIX command prompts and what you can do with them, see the next section of this guide.

Transferring Files

To transfer files, use your preferred SFTP client to connect to:

sftp.osc.edu

You may see warning message including SSH key fingerprint. Verify that the fingerprint in the message matches one of the SSH key fingerprint listed here, then type yes.

Since process times are limited on the login nodes, trying to transfer large files directly to oakley.osc.edu or other login nodes may terminate partway through. The sftp.osc.edu is specially configured to avoid this issue, and so we recommend it for all your file transfers.

Note: The sftp.osc.edu host is not connected to the scheduler, so you cannot submit jobs from this host. Use of this host for any purpose other than file transfer is not permitted.

Firewall Configuration 

See our Firewall and Proxy Settings page for information on how to configure your firewall to allow connection to and from OSC.

Setting up X Windows (Optional)

With an X Windows server you will be able to run graphical applications on our clusters that display on your workstation. To do this, you will need to launch your X Windows server before connecting to our systems. Then, when setting up your SSH connection, you will need to be sure to enable "X11 Forwarding".

For users of the command-line ssh client, you can do this by adding the "-X" option. For example, the below will connect to the Oakley cluster with X11 forwarding:

$ ssh -X username@oakley.osc.edu

If you are connecting with PuTTY, the checkbox to enable X11 forwarding can be found in the connections pane under "Connections → SSH → X11".

For other SSH clients, consult their documentation to determine how to enable X11 forwarding.

NOTE: The X-Windows protocol is not a high-performance one. Depending on your system and Internet connection, X Windows applications may run very slowly, even to the point of being unusable. If you find this to be the case and graphical applications are a necessity for your work, please contact OSC Help to discuss alternatives.
Service: 

Allocations and Accounts

The Ohio Supercomputer Center provides cycles to both academic and industrial clients. The methods for gaining access to the systems are different for each user community.

Primarily, our users are Ohio-based and academic, and the vast majority of our cycles will continue to be consumed by Ohio-based academic users. Those cycles are allocated to academic PIs via an allocations process overseen by the Statewide Users Group, which evaluates applications via a peer review process. There are two basic classes of academic allocations:

  1.  an allocation of a number of resource units (RUs) that never expire
  2.  an annual allocation process that awards an RU allocation for the upcoming calendar year. The annual allocation process is designed for "discovery level" users, who consistently consume large amounts of resources, and allow the Allocations Committee to better allocate the limited Center resources.

Other users (non-Ohio academic or industrial/commercial) interested in using Center resources may purchase cycles at a rate set by the expected total consumption. Expert consulting support is also available.

You may find more information about "RU" calculation and charging here.
 

  Ohio Academic Other
  Rolling Annual Rolling
Duration Unlimited 1 year Unlimited
Allocation Approval SUG SUG OSC Director
Call for Proposals At any time November At any time
  Apply [By invitation] Apply

For users interested in gaining access to larger resources, please contact OSC Help. We can assist you in applying for resources at an NSF or XSEDE site.

Once a project has been created, the PI can create accounts for users by contacting OSC Help to add existing users to that project

I need to add authorized users to my project

Please contact OSC Help. We require PI approval. 

I need additional resources for my existing project

If you already have a project, but are running low on allocated resources, you can click on the links above for your project type to request additional resources.

See our Project Application page for more information. 

I wish to use OSC to support teaching a class

We provide special classroom allocations for this purpose. You may apply here. See our Project Application page for more information. 

I don't think I fit in the above categories

Please contact us.

Managing Your Account

Once your account is set-up, you will receive a welcome letter from us containing your username and instructions for setting up your account.

MyOSC, one of our web portals, provides services to OSC users, including:

  • Password changes and resets for your HPC accounts
  • Changing your UNIX shell
  • Updating your email address
  • Tracking your project(s) balance​

Note: HPC password changes and resets take 3-5 minutes to process. If you attempt to login with your new password, you will receive an error. Too many failed login attempts results in your account being automatically locked for 5 minutes.

 

Are the answers you're looking for not here? Contact OSC Help

Service: 

Managing Your Project

Principal Investigators of OSC projects are responsible for updating their authorized user list, their outside funding sources, and their publications and presentations that cite OSC. All of these tasks can be accomplished by contacting OSC Help with the information. PIs are also responsible for monitoring their project's Resource Unit balance and for submitting a proposal to the Allocations Committee of the Statewide Users Group before the balance goes negative. Projects with negative balances are subject to restrictions, causing submitted jobs to have lower priority in the queue. We have a resource page for your convenience. 

Allocation managment for Ohio-based academic projects

Eligible Principal Investigators must be tenure-track faculty members or permanent research scientists at Ohio colleges or universities. All other researchers, such as postdoctoral researchers, graduate students, and undergraduate students, may be authorized users under a project led by an eligible PI.

Principal Investigators who plan to submit a proposal for additional resource units should have the following information ready when they fill out the application form:

  • names, departments, affiliations, and email addresses of new authorized users
  • names and email address of two or three suggested peer reviewers
  • document with performance reports and justification for request
  • electronic version of resume/CV
  • list of publications and presentations that cite OSC
  • proposal text
  • document with other pertinent information (optional)

Send a query to grant@osc.edu or to oschelp@osc.edu if you require more detailed information concerning the application process.

Once a proposal has been approved for peer review, any restrictions that have been accrued due to a negative balance will be lifted. The Allocations Committee meets every other month (even-numbered months), and awards are made after the meeting.

Account Maintenance and Usage: Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details. 

Allocation managment for purchased cycles

Please disregard any RU messages or instructions to apply for additional resources. If you have any billing questions, please consult your contact at OSC or inquire with OSC Help.

 

Service: 

Project Applications

Applying for Academic Projects

Statewide academic allocation of computing resources and services is a peer-reviewed account process governed by the Allocations Committee of the Statewide Users Group and subject to the Ohio Supercomputer Center (OSC) Code of Ethics for Academic Users.

The procedures described enable services to be available to approved projects conducted by eligible principal investigators (PIs) at Ohio academic institutions. In order to be an eligible PI at OSC, you must be eligible to hold PI status at your college, university, or research organization administered by an Ohio academic institution (i.e., be a full-time, permanent academic researcher or tenure-track faculty member at an Ohio college or unviersity). Students, post-doctoral fellows, visiting scientists, and others who wish to use the facilities may be authorized users on projects headed by an eligible PI. Once a PI has received their project information, he/she can contact OSC Help to add new authorized user(s) to a project at any time.

All first-time PIs must submit a signature page (electronically or hard copy). Please send a copy of the signature page to the attention of the Allocations Committee, OSC, 1224 Kinnear Road, Columbus, Ohio, 43212.

OSC's online project application is part of an electronic system that leads you through the process step by step. Before you begin to fill in the application form, especially if you are new to the process, look at the Project Application Help page. From this help page, please note the required materials for each request. Have all information and materials (suggested reviewer names and addresses, proposal text file names, etc.) at hand before you begin the application form. You can save a partially completed project application form for later use. As a reminder, you cannot re-submit application materials if they have already been used. 

Online Project and Account Application Form (NEW OSC PI)
Online Project Application Form (EXISTING PI)
Classroom Project Application Form (use our resources and software in the classroom)

If you need assistance, please contact OSC Help.

Applying for Commercial Access

Researchers from commercial organizations who wish to use the OSC's resources should complete the Ohio business request form or request access through the form for someone other than an Ohio business or academic researcher.

Add a new user

Please submit your request to OSC Help or use our online submission form. For reference, we will need the following:

Project:
Full Name:
If not U.S. Citizen, country of citizenship:
Institution:
Department:
Affiliation:
Email address:
Special Requests?

Letter of Commitment for Outside Funding Proposals

OSC will provide a letter of commitment users can include with their account proposals for outside funding, such as from the Department of Energy, National Institutes of Health, National Science Foundation (limited to standard text, per NSF policy), etc. This letter details OSC's commitment to supporting research efforts of its users and the facilities and platforms we provide our users. [Note: This letter does not waive the normal OSC account application process; it merely states that OSC is willing to support such application.] The information users must provide for the letter is:

  • address to which the letter should be directed (e.g. NSF, Department, mailing address)
  • name of funding agency's representative
  • name of the proposal, as well as names and home institutions of the Co-PIs
  • number of RUs per year you would apply for if you were to receive funding
  • number of years of proposed research
  • solicitation number

Send e-mail with your request for the commitment letter to OSC Help or submit online. We will prepare a draft for your approval and then we will send you the final PDF for your proposal submission. Please allow at least two working days for this service.

Letter of Support for Outside Funding Proposals

Letters of support may be subject to strict and specific guidelines, and may not be accepted by your funding agency.

If you need a letter of support, please see above "Letter of Commitment for Outside Funding Proposals". 

Applying at NSF Centers

Researchers requiring additional computing resources should consider applying for allocations at National Science Foundation Centers. For more information, please write to oschelp@osc.edu, and your inquiry will be directed to the appropriate staff member.:

We require you cite OSC in any publications or reports that result from projects supported by our services.

UNIX Basics

OSC HPC resources use an operating system called "Linux", which is a UNIX-based operating system, first released on 5 October 1991. Linux is by a wide margin the most popular operating system choice for supercomputing, with over 90% of the Top 500 list running some variant of it. In fact, many common devices run Linux variant operating systems, including game consoles, tablets, routers, and even Android-based smartphones.

While Linux supports desktop graphical user interface configurations (as does OSC) in most cases, file manipulation will be done via the command line. Since all jobs run in batch will be non-interactive, they by definition will not allow the use of GUIs. Thus, we strongly suggest new users become comfortable with basic command-line operations, so that they can learn to write scripts to submit to the scheduler that will behave as intended. We have provided some tutorials explaining basics from moving about the file system, to extracting archives, to modifying your environment, that are available for self-paced learning. We also occasionally offer training classes on this topic.

Service: 

Classroom Project Resource Guide

Welcome to OSC! 

This document includes some useful information to help your classroom use of our resources be effective.

Application: Please submit an application for a classroom project. We will require your resume/CV as the instructor and PI of the project, a class roster with email addresses, and a class syllabus. 

Access: We suggest that students consider connecting to our OnDemand portal to access the HPC resources. All production supercomputing resources can be accessed via that website, without having to worry about client configuration. We have a guide for new students to help them figure out the basics of using OSC; included are basics on getting connected, HPC system structure, file transfers, and batch systems.

Resources: We currently have three production clusters, Oakley, Ruby, and Owens, with Nvidia GPUs available. All systems have "debug" queues, during typical business hours, that allow small jobs of less than 1 hour to start much quicker than they might otherwise.

If you need to reserve access to particular resources, please contact OSC Help, preferably with at least two weeks lead time, so that we can put in the required reservations to ensure resources are available during lab or class times.

Software: We have a list of supported software, including sample batch scripts, in our documentation. If you have specific needs that we can help with, let OSC Help know.

Account Maintenance: Please use MyOSC to update your password, shell, and check usage on the project.

SupportHelp can be found by contacting OSC Help weekdays, 9AM to 5PM (1-800-686-6472 or 614-292-1800).
Fill out a request online. 

 

We regulary maintain our webpages to show relevant events at the center (including training) and system notices on our main page (osc.edu). We also provide important information in the “message of the day” (visible when you log in). Furthermore, you can receive notices by following @HPCNotices on Twitter. 
 

Useful Links:

FAQ: http://www.osc.edu/supercomputing/faq

Main Supercomputing pages: http://www.osc.edu/supercomputing/

HOWTO

Our HOWTO collection contains short tutorials that help you step through some of the common (but potentially confusing) tasks users may need to accomplish, that do not quite rise to the level of requiring more structured training materials. Items here may explain a procedure to follow, or present a "best practices" formula that we think may be helpful.

Service: 

HOWTO: Add python packages using the conda package manager

While our python installations come with many popular packages installed, you may come upon a case where you need an addiditonal package that is not installed.  If the specific package you are looking for is available from Anaconda.org (formerlly binstar.org) you can easily install it and required dependencies by using the Conda package manager.

To be able to install a package using the conda package manager:

  • You must use a Anaconda distribution of python:
    • On Oakley the following modules:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Ruby:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Owens:
      • python/2.7, python/3.5
  • * = latest suffix refers to a distribution that will be electively updated
  • Package should be available through Anaconda.org and/or using pip (see next section).

If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

But I use virtualenv and/or pip!

See the comparison to these package management tools here:

https://conda.io/docs/commands.html#conda-vs-pip-vs-virtualenv-commands

Pip installations are supported:

module load python
module list # check which python you just loaded
pip install --upgrade --user PACKAGE # where PACKAGE is a valid package name

If binaries exist in a pip installation they are usually installed in:

$HOME/.local/bin

The one issue with this approach is portability with multiple Python modules.  If you plan to stick with a single Python module, then this should not be an issue.  However, if you commonly switch between Python 2 and Python 3, then be aware of the potential trouble in using the same installation location for all Python versions.

Virtualenv may also work, but is not under active testing at OSC.

Procedure

We will install the yt package to a local directory in this example.

Load proper python module

module load python/2.7.8

Clone python installation to local directory

conda create -n local --clone="$PYTHON_HOME"

This will clone the entire python installation to ~/envs/local directory. The process will take serveral minutes.

conda create -n local

This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.

conda create -n local python={version} anaconda

If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=2.4" for version 2.4.

Activate clone environment

source activate local

On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

conda remove conda-build
conda remove conda-env

Install package

conda install yt
  • Replace yt with the name of the package you want to install, as listed by anaconda.org.
If there are errors on this step you will need to resolve them before continuing.

Test python package

Now we will test our installed python package by loading it in python and checking its location to ensure we are using the correct version.

python -c "import yt;print yt.__file__"

Output:

$HOME/.conda/envs/local/lib/python2.7/site-packages/yt/__init__.py
  • Replace both instances of yt  with the name of the package you installed.
Remember, you will need to load the proper version of python before you go to use your newlly installed package.  Packages are only installed to one version of python.

Install your own python modules

If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

Further Reading:

Conda Test Drive: https://conda.io/docs/test-drive.html 

HOWTO: Install your own python modules

While we provide a number of Python modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Python modules, and make them available inside of Python. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

Please consider using conda python package manager before you try to build python using the method explained here. We have an instruction on conda here.

Gather your materials

First, you need to collect up what you need in order to do the installation. To keep things tidy, we will do all of our work in $HOME/local/src . You should make this directory now.

mkdir -p $HOME/local/src

Now, we will need to download the source code for the module we want to install. In our example, we will use "NumExpr", a module we already provide in the system version of Python. You can either download the file to your desktop, and then upload it to OSC, or directly download it using the wget utility (if you know the URL for the file).

cd ~/local/src
wget http://numexpr.googlecode.com/files/numexpr-2.0.1.tar.gz

Now, extract the downloaded file. In this case, since it's a "tar.gz" format, we can use tar to decompress and extract the contents.

tar xvfz numexpr-2.0.1.tar.gz

You can delete the downloaded archive now, if you wish, or leave it around should you want to start the installation from scratch.

Build it!

Environment

To build the module, we will want to first create a temporary environment variable to aid in installation. We'll call it "INSTALL_DIR".

export INSTALL_DIR=${HOME}/local/numexpr/2.0.1

I am following, roughly, the convention we use at the system level. This allows us to easily install new versions of software without risking breaking anything that uses older versions. We have specified a folder for the program (numexpr), and for the version (2.0.1). Now, to be consistent with python installations, we're going to create a second temporary environment variable, which will contain the actual installation location.

export TREE=${INSTALL_DIR}/lib/python2.7/site-packages

Now, make the directory tree.

mkdir -p $TREE

Compile

To compile the module, we should switch to the GNU compilers. The system installation of Python was compiled with the GNU compilers, and this will help avoid any unnecessary complications. We will also load the Python module, if it hasn't already been loaded.

module swap intel gnu
module load python

Now, build it. This step may vary a bit, depending on the module you are compiling. You can execute python setup.py --help to see what options are available. Since we are overriding the install path to one that we can write to, and that fits our management plan, we need to use the --prefix option.

python setup.py install --prefix=$INSTALL_DIR

Make it usable

At this point, the module is compiled and installed in ~/local/numexpr/2.0.1/lib/python2.7/site-packages . Occasionally, some files will be installed in ~/local/numexpr/2.0.1/bin as well. To ensure Python can locate these files, we need to modify our environment.

Manual

The most immediate way - but the one that must be repeated every time you wish to use the module - is to manually modify your environment. If files are installed in the "bin" directory, you'll need to add it to your path. As before, these examples are for bash, and may have to be modified for other shells. Also, you will have to modify the directories to match your install location.

export PATH=$PATH:~/local/numexpr/2.0.1/bin

And, for the python libraries:

export PYTHONPATH=$PYTHONPATH:~/local/numexpr/2.0.1/lib/python2.7/site-packages

Hardcode it

We don't really recommend this option, as it is less flexible, and can cause conflicts with system software. But, if you want, you can modify your .bashrc (or similar file, depending on your shell) to set these environment variables automatically. Be extra careful; making a mistake in .bashrc (or similar) can destroy your login environment in a way that will require a system administrator to fix. To do this, you can copy the lines above modifying $PATH and $PYTHONPATH into .bashrc. Remember - test them interactively first! If you destroy your shell interactively, the fix is as simple as logging out and then logging back in. If you break your login environment, you'll have to get our help to fix it.

Make a module (recommended!)

This is the most complicated option, but it is also the most flexible, as you can have multiple versions of this particular software installed, and specify at run-time which one to use. This is incredibly useful if a major feature changes that would break old code, for example. You can see our tutorial on writing modules here, but the important variables to modify are, again, $PATH and $PYTHONPATH . You should specify the complete path to your home directory here, and not rely on any shortcuts like ~ or $HOME .  Below is a modulefile written in Lua:

If you are following the tutorial on writing modules, you will want to place this file in $HOME/local/share/modulefiles/numexpr/2.0.1.lua :

-- This is a Lua modulefile, this file 2.0.1.lua can be located anywhere
-- But if you are following a local modulefile location convention, we place them in
-- $HOME/local/share/modulefiles/
-- For numexpr we place it in $HOME/local/share/modulefiles/numexpr/2.0.1.lua
-- This finds your home directory
local homedir = os.getenv("HOME")
prepend_path("PYTHONPATH", 
pathJoin(homedir, "/local/numexpr/2.0.1/lib/python2.7/site-packages"))
prepend_path(homedir, "local/numexpr/2.0.1/bin"))

 

Once your module is created (again, see the guide), you can use your python module simply by loaded the software module you created.

module use $HOME/local/share/modulefiles/
module load numexpr/2.0.1
Service: 

HOWTO: Configure the MATLAB Parallel Computing Toolbox

Introduction

The MATLAB Parallel Computing Toolbox and Distributed Computing Server are designed to allow users to create and launch parallel MATLAB jobs on a cluster of compute nodes. It also allows users to remotely connect to OSC resources, whether to run parallel jobs in MATLAB or to use toolboxes for which users own their own licenses. This guide will explain the basics of how to configure your Parallel Computing Toolbox for OSC systems.

Versions

The following versions of the MATLAB Parallel Computing Toolbox are supported at OSC:

VERSION OAKLEY
R2013a X
R2013b X
R2014a X
R2014b X
R2015a X
R2015b X

Usage Overview

When you use the MATLAB Parallel Computing Toolbox, you have a MATLAB client session and one or more MATLAB workers. The client session may run on your laptop/desktop computer ("remote client") or it may run on an OSC login, OnDemand, or compute node. The MATLAB workers always run on the OSC cluster as part of a batch job. In the client session you will run MATLAB commands to set up and run a batch job, and MATLAB submits the job for you.

This document describes how to perform a computation on a single worker (one processor) or on multiple workers using a script with a "parfor" loop.

Licensing issues

You must have a license for MATLAB and the MATLAB PCT to allow you to run the MATLAB client session on your laptop/desktop computer. OSC has licenses for the MATLAB Distributed Computing Server, which covers the MATLAB workers running on the cluster.

OSC is included in the OSU site license for MATLAB, so OSC users have the option of running the client session on an OSC machine. You can manage and view your MATLAB licenses using the Mathworks License Center.

In any case, in order to use MDCS license on OSC, you need to be added to the license server. So, please contact OSC Help

Remote client

If you run your MATLAB client session on your local computer, it is considered a remote client. You will be able to use any toolboxes that you have a license for by submitting batch jobs from your MATLAB client. The batch jobs you submit from your MATLAB client through the PCT will be able to utilize functions such as "parfor" and "spmd" to run your code in parallel, in addition to being able to run normal MATLAB code.

Client running on Oakley

Through the MATLAB installations on Oakley, you will be able to take advantage of the interactive capabilities in the MATLAB PCT, as well as submitting batch jobs through the MATLAB client sessions. OnDemand or VNC is needed to use this (the MATLAB PCT is GUI only at this time), but it will allow you to debug your parallel jobs using various tools in real time. Jobs submitted through the client sessions on Oakley behave the same as jobs submitted by qsub, but can utilize the PCT functions such as "parfor" and "spmd".

Performance limitations

Parallel MATLAB on a cluster has a lot of overhead. It may not give you the speedup you expect, especially if you're solving small problems. Another consideration is that the MATLAB workers are single-threaded. They don't take advantage of the multithreading built into many MATLAB functions.

Download and Install the Configuration Files

The first step is to download the necessary configuration files to the computer where you will be running the MATLAB client session.

    Click the link below to download the files.

    OSCMatlabPCT

    OSCMatlabPCT Configuration Resources -- Directory Structure

    The OSC MATLAB PCT Configuration package contains the following files and directories:

    OSCMatlabPCT (top-level directory)

    • config - A directory containing the cluster profiles and configuration functions for using the MATLAB PCT at  OSC.  Note: the only function that users should edit is the "addSubmitArgs" function, as this allows setting some additional batch options, such as walltime, email, and custom job name.  All the other functions have been specially prepared by OSC staff to allow MATLAB to work in harmony with the HPC system on Oakley.
    • launch - A directory containing two scripts, "client_session_script.m" for launching jobs and "reconnect_client_session.m" for reconnecting to a job.  These scripts illustrate key concepts of using the PCT when accessing our system remotely via the MATLAB client on your personal computer.  They also apply to submitting batch jobs from a MATLAB client running in OnDemand.  Both scripts are heavily commented and give usage details beyond what this document can provide.  Note that these scripts contain commands to be run in your client session; do not submit them using the batch command.
    • PCTtestfiles -  A directory containing parfor use-cases. The first, "eigtest", is an example of how to program an extremely simple entry function using parfor.  It simply computes the eigenvalues of multiple large, random matrices.  The second case, is an example of how to run a parallel Simulink simulation using parfor.  The entry function is "paralleltestv2", which calls the function "parsim" inside a parfor loop.  The "parsim" script contains commands that initialize Simulink and run the simulation on each of the parfor workers.

    Import and Configure the Cluster Profile

    These configuration steps need to be done once before you use PCT.  (If you upgrade to a new version of MATLAB you'll have to reconfigure.)  There are also command line options for performing these tasks.

    1. After launching MATLAB, click the "Parallel" dropdown menu from the "Environment" menu and select "Manage Cluster Profiles".  At this time, a new window should open displaying the Cluster Profile Manager.  


    Image of MATLAB Parallel Menu


    2. In the Cluster Profile Manager window, click the "Import" button and locate the "clusterProfiles" directory contained within the OSCMatlabPCT/config/clusterProfiles directory.  

    Cluster profiles are named according to filesystem configuration and version.  Select the file which corresponds to your current version of MATLAB and your filesystem configuration.  If you're running remotely on your laptop/desktop computer select a "NonShared" version.  If you're running on Oakley, select a "Shared" version.  Click "Open" to proceed.  


    Screen shot 2014-06-06 at 11.14.32 AM.png


    3. If you are using a shared filesystem configuration: No further modifications will need to be made to the cluster profile, and you can close the Cluster Profile Manager and continue with configuring your job. 

    If you are using a non-shared filesystem configuration: You will need to make some changes to your cluster profile before exiting the Cluster Profile Manager.  Click the "Edit" button to enable editing.


    Screen shot 2014-06-06 at 11.36.41 AM.png


    4. In the editing window under "Submit Functions", you should see two entries -- IndependentSubmitFcn and CommunicatingSubmitFcnIntel.  These fields contain cell arrays with three values: a handle to the submit function, the hostname of the cluster, and the remote job storage location where results files and logs will be stored.  You only need to change the remote job storage location.  Use an absolute path to a location in your OSC home directory where you'd like the output of your job (or jobs) to be retained.  


    Image of MATLAB Cluster Profile Properties

    Here is an example of the full syntax:

    {@independentSubmitFcn, 'oakley.osc.edu', '/my/home/dir/MATLAB'}

    {@communicatingSubmitFcnIntel, 'oakley.osc.edu', '/my/home/dir/MATLAB'}

    Note:  If you try to validate your configuration, the last test will always fail for remote clients, even if the configuration is correct.


    5. Include the configuration files in your MATLAB path.  Click on "Set Path", then "Add Folder".  Locate and select the OSCMatlabPCT\config directory.  Click "Select Folder" and "Save".  (Alternatively you can use the "addpath" command in the MATLAB command window.)


    Configure and Run a Batch Job

    The primary way to utilize the MATLAB PCT at OSC is to submit batch jobs from the MATLAB client, either on your local PC or on Oakley via OnDemand.  These instructions assume that you have already configured your MATLAB client as described above.

    IMPORTANT: MATLAB treats the output directory specified with the submit functions as SCRATCH file space, and will cleanup this directory after you have successfully retrieved your data.  However, in practice it has been observed that MATLAB sometimes cleans up this directory even if the commands are unsuccessful (such as in the case of a large file transfer).  To avoid data loss, please make sure your entry function/script copies any important output files to an alternate location for redundancy.  Or simply save your data to a directory of your choice.
    1. Write your script to be run on the cluster.  This can be just a serial script or it may contain parallel commands such as "parfor".  Some example scripts are provided in the directory OSCMatlabPCT/PCTtestfiles.  The example below uses "eigtest.m" as the script to be run.
    2. In your client session:  Using "client_session_script.m" as a guide, run the commands to connect to the cluster and launch a batch job.  We don't recommend that you run the script as written, at least at first.  Until you're familiar with using the PCT, simply copy commands out of the script and paste them into your MATLAB command window as needed.  There are two important functions involved in launching a job.  The command "parcluster" creates a cluster object in the MATLAB workspace, which is used to store information sent between the remote cluster and your MATLAB client.  The command "batch" begins an automated process that can connect to the Oakley cluster, submit a job to PBS, and initialize the MATLAB Distributed Computing Server (MDCS).  For more on the batch command, see the "client_session_script.m" file, and the MATLAB help.
    3. When your job has been successfully configured, you will be prompted for your username and password at OSC.  After entering this, your job will be submitted to the system, and assigned an OSC job ID.  Please note: as explained in the "client_session_script" example, MATLAB remembers your jobs on the cluster by the job index number, not the OSC job ID.  Also, if you are unable to qstat your job immediately after it is submitted, don't worry!  There seems to be some latency between the MATLAB MDCS and the batch system when it comes to exchanging job status info, but this should not affect your ability to track your job.  In fact, as shown in the example, there are several ways to get information about your jobs directly from your MATLAB client. After submission you can check your job's progress or retrieve results at any time (even after closing the MATLAB client), and you can continue using the MATLAB client for other work if you wish.
    4. Wait for your job to complete.  You may use the "wait" command in your client session or simply monitor your job's status with "qstat" on an Oakley login node to know when your job has completed.
    5. Reconnect after closing your MATLAB client session.  The script "reconnect_client_session.m" illustrates how to find your job information again if you restart your MATLAB client session.
    6. Retrieve your results.  The "load" or "fetchOutputs" command retrieves the results of your computations from your MATLAB working directory on Oakley after the job has finished.  They also report errors if your run failed.  "Load" is used for exporting the workspace from an entry script, and "fetchOutputs" is used for retrieving the specified output arguments of an entry function.  For large data, there is an important caveat: MATLAB does not utilize sftp when fetching your files, so large files might take a long time to retrieve.  Additionally, if your output arguments/workspace is over a certain size (around 2GB), using the "load" or "fetchOutputs" commands will give an "index out of range" error caused by the MDCS failing to save the output to file.  This is due to an internal limitation in the default version of MATLAB's ".mat" files.  However, you may work around this limitation by manually saving your workspace or arguments in your entry script, using the '-v7.3' switch in the "save" command.

    Specifying Additional PBS Options for Your Batch Job

    The submit functions are MATLAB functions which serve two primary purposes: to create the appropriate connections to the cluster where your job will run, and to procedurally generate and submit a shell script to the queue on Oakley.  Your nodes and cores (ppn) are determined by the number of MATLAB workers you request in your submit script, however to change or add additional arguments or resource requests you must add the PBS commands to the string in the function "addSubmitArgs" (located in config).  The default arguments in the string are for a walltime of 1:00:00, a job name "default", and emails sent on abort, begin, and end of the job.  This function is called in the submit functions, so advanced users may choose to do what they wish with how this file is called in the submit functions to streamline their workflow if multiple configurations are needed.

    Configure and Run an Interactive Job

    There are a couple useful ways for OSU users to use the interactive features of the MATLAB PCT.  Because of our license terms, this option is available only to OSU faculty, staff, and students.

    Using OnDemand with a Shared Cluster Profile

    OSU users may take advantage of the MATLAB licenses on Oakley to submit interactive "parpool" (R2013b or later) or "matlabpool" (R2013a) jobs the batch system.  These MATLAB PCT functions allow you to run parallel functions such as "parfor" or "spmd" directly from command line or a script, and allow users to debug or interact with parallel jobs in real time.  OSC offers this functionality in MATLAB clients running through the OnDemand portal.  In this method, simply start on Oakley desktop session in OnDemand, open a terminal window, load the correct module, and start MATLAB.  Be sure the OSCMatlabPCT/config directory is in your path.  You will then import the correct shared cluster profile for your version of MATLAB, and use the "parpool" (R2013b or later) or "matlabpool" (R2013a) command to initialize an automatic parallel pool. This might take a while to set up, as the pool request must go throught the batch system like everything else, so be patient!  After a couple minutes or more (depending on the resource request), your pool will connect to the MATLAB workers, and you may begin running parallel commands. This will allow simulation of how jobs will behave when submitted to the MDCS as a batch job.

    Using VNC with the "local" Cluster Profile

    Another way to run an interactive parallel job using the MATLAB PCT is to submit an Interactive Batch job from using qsub, and run MATLAB on a compute node using VNC to utilize the GUI.  You can then import the "local" cluster profile, which runs the workers directly on the node.  Only one node can be used, but workers will run on multiple processors on the node.  This can be used in situations when using OnDemand is not ideal, but users should carefully follow the directions provided so as to not leave VNC processes behind on the node.  Directions for using VNC in a batch job are located here: https://osc.edu/documentation/howto/use-vnc-in-a-batch-job">https://osc.edu/documentation/howto/use-vnc-in-a-batch-job.

    Additional Documentation

    www.mathworks.com/index.html?s_tid=gn_logo">http://www.mathworks.com/index.html?s_tid=gn_logo">Mathworks has written extensive documentation covering all of the public functions within the Parallel Computing Toolbox. The PCT homepage is located at www.mathworks.com/products/parallel-computing/">http://www.mathworks.com/products/parallel-computing/">The Mathworks Parallel Computing Toolbox Homepage.

    For more information about how to construct and run an independent job, see the Mathworks documentation page "www.mathworks.com/help/distcomp/program-independent-jobs.html">http://www.mathworks.com/help/distcomp/program-independent-jobs.html">Program Independent Jobs".  

    For more information about how to construct and run a communicating job, see the Mathworks documentation page "www.mathworks.com/help/distcomp/introduction.html">http://www.mathworks.com/help/distcomp/introduction.html">Program Communicating Jobs".

    Errors

    If you have any errors related to the OSC MATLAB PCT configuration files, or have any other questions about utilizing the MATLAB PCT at OSC please contact OSC Help with: your user id, any relevant error messages, a job ID(s) if applicable, and the version of MATLAB you are using.

    HOWTO: Connect to OSC services using OSC Connect

    To connect to OSC services, a secure tunnel to a session is required. This can be done relatively simply in OSX and Linux by using the SSH functionality built into the system, but Windows users have had to configure and use third party applications like PuTTY or Java to access secure resources at OSC. OSC Connect is a native windows application written in C# and compiled for .NET 2.0, providing preconfigured management of secure tunnel connections for Windows users, as well as providing a launcher for secure file transfer, VNC, terminal, and web based services. 

    OSC Connect is supported on Windows versions from Windows XP through Windows 10.

    Getting Started

    1. Download the latest release of OSC Connect from here: Latest Build
      • Click OSCConnect.exe  and download. Use "Save link as" to download this file to a folder of your choice. 
      • See here for all the previous releases. 
    2. Double-Click the OSCConnect.exe icon to run the application. In the current state, OSC Connect is entirely deployed by a single executable file; no further installation is required.   ​
    3. When you run OSC Connect, a temporary folder named "ConnectFiles" with four additional files will be created. These are required for proper functionality of the application. Please ensure that these files are permitted by your IT administrator:
      • plink.exe is the command-line version of PuTTY used by the application to create the secure connection to OSC resources.
      • putty.exe  is the GUI application of PuTTY used to provide terminal emulation remote console connections to OSC resources.
      • vncviewer.exe is the VNC viewer client used to view a remote desktop session.
      • WinSCP.exe  is the SFTP client used for file transfer.
    4. Once your connections to OSC services as well as the OSC Connect app is closed, the temporary folder named "ConnectFiles" will be removed automatically. 

    OSC Connect Graphical User Interface

    After you double-click the  OSCConnect.exe  icon oscconnect_icon , the application graphical user interface is shown as below:

     interface

    • Network Status: it indicates which OSC cluster you will be connected to. The option can be changed in "Settings". 

    • Settings: it provides several configuration options to modify the behavior of the application. 

      • Connection Settings: use this dropdown to select the default host/cluster. Selecting a server here will change the endpoint for tunneling, sftp connections, console connections, and connectivity checking.

      • System Settings

        • Detect Clipboard Activity: when this option is enabled, the application will detect valid data on the Windows clipboard and populate the application. ( Default: Off )

        • Check for New Versions: when this option is enabled, the application will check for version updates. (Default: on)

      • Automation Settings:

        • Save User Credentials: when this option is enabled, it allows the application to remember the user when the application is reopened. This saves the user credentials to the user settings using DPAPI Encryption. Passwords are decrypted only by current the Windows user account. ( Default: Off )

        • Launch Tunnel On Import: when this option is enabled, the tunnel will automatically connect when the application detects a valid clipboard string and the user credentials have been entered. ( Default: On )

      • VNC Settings

    • ​About: it provides additional information about OSC Connect.

    After you provide your OSC Credentials, i.e. your OSC HPC username and password, more functionalities are available as shown below:

    gui2

    • SFTP: it uses WinSCP as the SFTP client to securely connect to the OSC file system over SFTP.
    • Console: it uses PuTTY to provides you shell access to OSC cluster.

    In addition, Session Type is provided such that advanced users can connect to a running session mannually.

    For most users, you can ingore Session Type and are able to connect to a running session through OSC OnDemand using the configuration generated after your session starts. See "Connecting to a Session" for more details. 

    Connecting to a Session

    The OSC Connect application can be used to connect to a running session launched through OSC OnDemand.

    OSC Connect does not have the ability to launch a new session
    1. Navigate to OSC Demand to create a new session under either "Desktops" or "Desktop Apps". 
      • More information on how to use OSC OnDemand can be found here
    2. Once your session is running, you have two options to connect to the session: "noVNC Instructions" and "Native Client Instructions". Click "Native Client Instructions" to use OSC Connect. 
    3. Follow the steps below to connect to the running session:
      • Step 1: Download the latest OSC Connect if you don't already have it. See "Getting Started" for more details. 
      • Step 2: Launch OSC Connect. 
      • Step 3: Click - osc://xxxxx. osc://xxxxx is a custom URI scheme that is registered when you launch the application. Simply click the link to populate the configuration information and connect to your running session. If OSCConnect.exe is not running when you click the URI, the OSC Connect application will be popped up. Enter your OSC HPC username and password and you will be able to connect to the session by clicking the "Connect" button.
      • For advanced users, you can skip step 3 and connect to a running session mannually. Depending on the type of application, you can choose either "iHPC VNC Desktop" to create a VNC tunnel or "Web Browser Session" to create a browser tunnel. 
    Be sure to launch OSCConnect.exe at least once before you use it to connect to a running session. The initial launch will add a key to your user registry that initializes the URI scheme.  

    FAQ

    I've clicked the osc:// link and nothing happened.

    Be sure to run OSCConnect.exe at least once. The initial launch will add a key to your user registry that initializes the URI scheme. If you move or rename the OSCConnect.exe file, you will need to run the application again manually to update the path in the handler.

    I've received the error "Unable to open helper application. The protocol specified in this address is not valid."

    This issue appears in some earlier versions of Internet Explorer when attempting to launch the application from a Temporary location. Download and run the OSCConnect.exe application, being sure to save the file to a non-temporary location.

     

     

    Service: 

    HOWTO: Identify users on a project account and check status

    An eligible principal investigator (PI) heads a project account and can authorize/remove user accounts under the project account (please check our Allocations and Accounts documentation for more details). This document shows you how to identify users on a project account and check the status of each user. 

    Identify Users on a Project Account

    If you know the project acccount

    If you know the project account (projectID), the following command will list all users on the project:

    getent group projectID
    

    The returned information is in the format of:

    projectID:*:gid: list of user IDs
    

    gid is the group identifier number unique for the project account projectID. 

    For example, the command  getent group PZS0530  lists all users on the project account PZS0530 as below:

    bash-4.1$ getent group PZS0530
    PZS0530:*:2959:yli,osc0539,elton,ananth,osc0413,dhudak,osc0695,ksaantha,changlee,buss,osc0414,osc0478,nsharma,kmanalo,nwadih,bsmith,...
    

    If you don't know the project acccount, but know the user account

    If you don't know the project account, but know the user account, userID, use the  groups  command to list all of the groups the userID belongs to:

    groups userID
    

    The returned information is in the format of:

    userID : list of groups
    

    where the first item after " userID : " is the primary group, i.e. the project account (projectID) which userID is on. Once you know the project account, use the command  getent group projectID  as discussed in the previous session to list all users on this project.

    For example, with the userID as xwang, the command groups xwang  returns the information as below:

    bash-4.1$ groups xwang
    xwang : oscgen lstc gaussian ...
    

    It lists all groups xwang belongs to, and oscgen is the project account which xwang is on. The command  getent group oscgen  lists all user accounts on the project account oscgen:

    bash-4.1$ getent group oscgen
    oscgen:*:200:jordan,jpu,mlewis,rmonroe,rmarshal,spears,sengupta,jtm,adraghi,lepage,ian,karin,remote,njustice,bedford,hjiang,tom,mudronja,elainep,gisadmin,airani,guilfoos,osc0498,osc0722,ngagnet,mfaerman,justinw,arya,mattm,echong,rahmed,jwright,...
    

    If you don't know either the project acccount or user account

    If you don't know either the project account or user account, you can use ldapsearch  command to get the user account based on the user's registration information such as name, email address, etc. 

    Use the following command to list all of the user accounts accociated with the name First Last:

    ldapsearch -x -LLL "(gecos=First Last)" | grep cn | awk '{print $2}'

    Use the following command to list all of the user accounts accociated with the email address email@address:

    ldapsearch -x -LLL "(mail=email@address)" | grep cn | awk '{print $2}'

    For example, with user's first name as Summer and last name as Wang, the command

    ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}' returns the information as below:

    bash-4.1$ ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}'
    xwang
    xwangnd
    ... 
    

    With user's email address as xwang@osc.edu, the command ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}' returns the information as below:

    bash-4.1$ ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}'
    xwang
    xwangnd
    ...
    

    Once you know the user account userID, follow the discussions in previous session to get all user accounts on the project. Please contact OSC Help if you have any questions. 

    Check the Status of a User

    Use the  finger  command to check the status of  a user account userID as below:

    finger userID
    

    For example, if the userID is xwang, the command  finger xwang  will return:

    bash-4.1$ finger xwang
    Login: xwang                            Name: Summer Wang
    Directory: /nfs/17/xwang                Shell: /bin/bash
    On since Tue Apr 19 09:58 (EDT) on pts/190 from localhost:43.0
    Mail forwarded to xwang@osc.edu
    No mail.
    No Plan.
    
    • The home directory of xwang is on /nfs/17 ( Directory: /nfs/17/xwang )
    • The shell of xwang is bash ( Shell: /bin/bash ). If the information is Shell:/access/denied , it means this user account has been either archived or resticted. Please contact OSC Help if you'd like to reactivate this user account.
    • xwang@osc.edu is the associated email with the user account xwang; that is, all OSC emails related to the account xwang will be sent to xwang@osc.edu ( Mail forwarded to xwang@osc.edu ). Please contact OSC Help if the email address asscoiated with this user account has been changed to ensure important notifications/messages/remindars from OSC may be received in a timely manner.

    Check the Usage and Quota of a User's Home Directory/Project's Project Space

    All users see their file system usage staticstics when logging in, like so:

    As of 2016 Apr 20 04:02 userid userID on /nfs/nn used XGB of quota 500GB and Y files of quota 1000000 files

    The information is from the file /usr/local/quotas/quota_report.txt, which is updated daily. Some users may see multiple lines asscoiated with userid, as well as information on project space usage and quota, if there is one. The usage and quota of the home diretory of userID is provided by the line including the file server your home directory is on (for more information, please visit Home Directories), while others (generated due to file copy) can be safely ingored. 

    You can check any user's home directory or a project's project space usage and quota by running:

    grep <userID OR projectID > /usr/local/quotas/quota_report.txt
    
    Service: 

    HOWTO: Install Local R Packages

    This document shows you the steps to install R packages locally without root access on OSC's Oakley cluster. 

    R comes with a single library  $R_HOME/library which contains the standard and recommended packages. This is usually in a system location. On Oakley cluster, it is   /usr/local/R/3.0.1/lib64/R/library . R also has a default value for a directory where users can install their own R packages. On Oakley cluster, it is ~/R/x86_64-unknown-linux-gnu-library/3.0 if the default R-3.0.1 module is loaded. This directory doesn't exist by default. The first time a user installs an R package, R will ask the user if s/he wants to use the default location and if yes, will create the directory.

    A Simple Example

    First you need to load the module for R:

    module load R
    

    On Oakley, the default R module is version 3.0.1 .

    Then fire up an R session:

    R
    

    To install package lattice, use this command inside R:

    > install.packages("lattice", repos="http://cran.r-project.org")
    

    It gives a warning: 

    Warning in install.packages("lattice") :
    'lib = "/usr/local/R/3.0.1/lib64/R/library"' is not writable
    Would you like to create a personal library
    ~/R/x86_64-unknown-linux-gnu-library/3.0
    to install packages into?  (y/n) 
    

    Answer y , and it will create the directory and install the package there.

    Setting the Local R Library Path

    If you want to use another location rather than the default location, for example, ~/local/R_libs/ ,  you need to create the directory first:

    mkdir ~/local/R_libs
    

    Then type the following command inside R:

    > install.packages("lattice", repos="http://cran.r-project.org", lib="~/local/R_libs/")
    

    It is a bit of burden having to type the long string of library path every time. To avoid doing that, you can create a file .Renviron in your home directory, and add the following line to the file:

    export R_LIBS=~/local/R_libs/ 
    

    Whenever R is started, the directory ~/local/R_libs/ is added to the list of places to look for R packages and so:

    > install.packages("lattice", repos="http://cran.r-project.org")
    

    will have the same effect as the previous install.packages() command. 

    To see the directories where R searches for libraries, use the command:

    >.libPaths();
    
    

    Setting The Repository

    When you install an R package, you are asked which repository R should use. To set the repository and avoid having to specify this at every package install, create a file .Rprofile in your home directory. This is the start up code for R. Add the following line to the file:

    cat(".Rprofile: Setting R repository:")
    repo = getOption("repos") 
    # set up the server from which you will download the package.
    repo["CRAN"] = "http://cran.case.edu" 
    options(repos = repo)
    rm(repo)
    

    Now you only need to do 

    > install.packages("lattice")
    

    That will download the package lattice from  http://cran.case.edu and install it in ~/local/R_libs  .

    Updating Packages

    > update.packages() inside an R session is the simplest way to ensure that all the packages in your local R library are up to date. It downloads the list of available packages and their current versions, compares it with those installed and offers to fetch and install any that have later versions on the repositories.

    Removing packages

    > remove.packages("lattice") inside an R session to remove package lattice . An even easier way is just to go into the directory ~/local/R_libs and remove the directory lattice from there.

     

    References

    Add-on packages in R installation guide (http://cran.r-project.org/doc/manuals/R-admin.pdf)

    Service: 

    HOWTO: Install your own Perl modules

    While we provide a number of Perl modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Perl modules. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

    CPAN Minus

    CPAN, the Comprehensive Perl Achive Network, is the primary source for publishing and fetching the latest modules and libraries for the Perl programming language. The default method for installing Perl modules using the "CPAN Shell", provides users with a great deal of power and flexibility but at the cost of a complex configuration and inelegant default setup.

     

    Setting Up CPAN Minus

    To use CPAN Minus, we must first load it, if it hasn't already been loaded. Note that this is not necessary if you loaded a version of Perl with the module load command.

    module load cpanminus
    

    Next, in order to use cpanminus, you will need to run the following command only ONCE:

    perl -I $CPANMINUS_INC -Mlocal::lib
    

     

    Using CPAN Minus

    In most cases, using CPAN Minus to install modules is as simple as issuing a command in the following form:

    cpanm [Module::Name]
    

    For example, below are three examples of installing perl modules:

    cpanm Math::CDF
    cpanm SET::IntervalTree
    cpanm DB_File
    

     

    Testing Perl Modules

    To test a perl module import, here are some examples below:

    perl -e "require Math::CDF"
    perl -e "require Set::IntervallTree"
    perl -e "require DB_File"
    

    The modules are installed correctly if no output is printed.

    What Local Modules are Installed in my Account?

    To show the local modules you have installed in your user account:

    perldoc perllocal
    

    Reseting Module Collection

    If you should ever want to start over with your perl module collection, delete the following folders:

    rm -r ~/perl5 
    rm -r ~/.cpanm
    

     

    Service: 

    HOWTO: Locally Installing Software

    Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space.

    NOTE: Throughout this document we'll assume you're installing into your home directory, but you can follow the steps below in any directory for which you have read/write permissions.
    This document assumes you are familiar with the process of building software using "configure" or via editing makefiles, and only provides best practices for installing in your home directory.

    Getting Started

    Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:

        local
        |-- src
        |-- share
            `-- lmodfiles
    

    This structure is how OSC organizes the software we provide. Each directory serves a specific purpose:

    • local - Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
    • local/src - Stores the installers -- generally source directories -- for your software. Also, stores the compressed archives ("tarballs") of your installers; useful if you want to reinstall later using different build options.
    • local/share/lmodfiles - The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.

    You can create this structure with one command.

    NOTE: Ensure $HOME is the full path of your home directory. You can identify this from the command line with the command echo $HOME.

    After navigating to where you want to create the directory structure, run:

        mkdir -p $HOME/local/src $HOME/local/share/lmodfiles
    

    Installing Software

    Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

    First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/local/src:

        cd $HOME/local/src
        wget https://github.com/git/git/archive/v2.9.0.tar.gz
    

    Now extract the tar file:

        tar zxvf v2.9.0.tar.gz
    

    Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/local/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

    You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

    Then, we can build Git using the following commands:

        cd git-2.9.0
        autoconf # this creates the configure file
        ./configure --prefix=$HOME/local/git/2.9.0 CC=icc CFLAGS=-static-intel
        make && make install
    

    Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

    Creating a Module

    Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

    We will be useing the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

    -- Local Variables
    local name = "git"
    local version = "2.9.0"
    
    -- Locate Home Directory
    local homedir = os.getenv("HOME")
    local root = pathJoin(homedir, "local", name, version)
    
    -- Set Basic Paths
    prepend_path("PATH", pathJoin(root, "bin"))
    prepend_path("MANPATH", pathJoin(root, "share/man"))
    

    NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

    Oakley, Ruby, and Owens use a Lua based module system. However, there is another module system based in TCL that will not be discussed in this HOWTO.                                                                         NOTE: TCL is cross-compatible and is converted to Lua when loaded. More documentation is available at https://www.tacc.utexas.edu/research-development/tacc-projects/lmod/ or by executing module help.

    Initializing Modules

    Any module file you create should be saved into your local lmodfiles directory ($HOME/local/share/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

    In the case of our Git example, you should create the directory $HOME/local/share/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

    To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/local/share/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using either module load git or module load git/2.9.0

    NOTE: module use$HOME/local/share/lmodfiles and module load "software_name" needs to be entered into the command line every time you enter a new session on the system.

    If you install another version later on (lets say version 2.9.1) and want to created a module file for it, you need to make sure you call it 2.9.1.lua. When loading Git, lmod will automatically load the newer version. If you need to go back to an older version, you can do so by specifying the version you want: module load git/2.9.0.

    To make sure you have the correct module file loaded, type which git which should give the output "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

    To make sure the software was installed correctly and that the module is working, type git --version which shuold give the output "git version 2.9.0".

    Further Reading

    For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.

    Service: 

    HOWTO: Reduce Disk Space Usage

    This "how to" will demonstrate how to lower ones' disk space usage.  The following procedures can be applied to all of OSC's file systems.

    We recommend users regularly check their data usage and clean out old data that is no longer needed.

    Users who need assistance lowering their data usage can contact OSC Help.

    Preventing Excessive Data Usage Before It Starts

    Users should ensure that their jobs are written in such a way that temporary data is not saved to permanent file systems, such as the project space file system or their home directory.  
     

    If your job copies data from the scratch file system or its node's local disk ($TMPDIR) back to a permanent file system, such as the project space file system or a home directory, you should ensure you are only copying the files you will need later.

    Identifying Old and Large Data

    The following commands will help you identify old data using the find command.

    find commands may produce an excessive amount of output.  To terminate the command while it is running, click  CTRL + C .

    Find all files in a directory that have not been accessed in the past 100 days:

    This command will recursively search the users home directory and give a detailed listing of all files not accessed in the past 100 days.  

    The last access time atime is updated when a file is opened by any operation, including grep , cat , head , sort , etc.

    find ~ -atime +100 -exec ls -l {} \;
    
    • To search a different directory replace ~ with the path you wish to search.  A period . can be used to search the current directory.
    • To view files not accessed over a different time span, replace 100 with your desired number of days.
    • To view the total size in bytess of all the files found by find , you can add  | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:
    find ~ -atime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'
    

    Find all files in a directory that have not been modified in the past 100 days:

    This command will recursively search the users home directory and give a detailed listing of all files not modified in the past 100 days.

    The last modified time mtime is updated when a file's contents are updated or saved.  Viewing a file will not update the last modified time.

    find ~ -mtime +100 -exec ls -l {} \; 
    
    • To search a different directory replace  ~  with the path you wish to search.  A period  .  can be used to search the current directory.
    • To view files not modified over a different time span, replace  100  with your desired number of days.
    • To view the total size in bytes of all the files found by find , you can add  | awk '{s+=$5} END {print "Total SIZE (bytes): " s}'  to the end of the command:
    find ~ -mtime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'
    

    List files larger than a specified size:

    Adding the -size <size>  option and argument to the find command allows you to only view files larger than a certain size.  This option and argument can be added to any other find command.

    For example, to view all files in a users home directory that are larger than 1GB:

    find ~ -size 1G -exec ls -l {} \;
    

    Deleting Identified Data

    CAUTION: Be careful when deleting files.  Be sure your command will do what you want before running it.  Extra caution should be used when deleting files from a file system that is not backed up, such as the scratch file system.

    If you no longer need the old data, you can delete it using the rm command.

    If you need to delete a whole directory tree (a directory and all of its subcontents, including other directories), you can use the rm -R command.  

    For example, the following command will delete the data directory in a users home directory:

    rm -R ~/data
    

    If you would like to be prompted for conformation before deleting every file, use the -i option.

    rm -Ri ~/data 
    

    Enter y or n when prompted.  Simply pressing the enter button will default to n .

    Deleting files found by find

    rm can be combined with any find command to delete the files found.  The syntax for doing so is:

    find <location> <other find options> -exec rm -i {} \;
    

    Where <other find options> can include one or more of the options  -atime <time> , -mtime <time> , and -size <size> .

    The following command would find all files in the ~/data directory 1G or larger that have not been accessed in the past 100 days, and then prompt for confirmation to delete each file:

    find ~/data -atime +100 -size 1G -exec rm -i {} \;
    

    If you are absolutely sure the files identified by find are okay to delete you can remove the -i option to rm and you will not be prompted.  Extreme caution should be used when doing so!

    Archiving Data

    If you still need the data but do not plan on needing the data in the immediate future, contact OSC Help to discuss moving the data to a archive file system.  Requests for data to be moved to the archive file system should be larger than 1TB. 

    Compressing

    If you need the data but do not access the data frequently, you should compress the data using tar or gzip.

    Moving Data to a Local File System

    If you have the space available locally you can transfer your data there using sftp or Globus

    Globus is recommended for large transfers.

    The OnDemand File application should not be used for transfers larger than 1GB.

    Service: 

    HOWTO: Submit multiple jobs using parameters

    Often users want to submit a large amount of jobs all at once with each using different parameters for each job.  These parameters could be anything, including the path of a data file or different input values for a program.  This how-to will show you how you can do this using a simple python script, a CSV file, and a template script.  You will need to adapt this advice for your own situation.

     

    Consider the following batch script:

    #PBS -l nodes=1:ppn=12
    #PBS -l walltime=80:00:00
    #PBS -n week42_data8
    
    # Copy input data to the nodes fast local disk 
    cp ~/week42/data/source1/data8.out $TMPDIR
    cd $TMPDIR
    
    # Run the analysis 
    full_analysis data8.in data8.out
    
    # Copy results to proper folder
    cp  data8.out ~/week42/results

    Lets say you need to submit 100 of these jobs on a weekly basis.  Each job uses a different data file as input.  You recieve data from two different sources, and thus your data is located within two different folders.  All of the jobs from one week need to store their results in a single weekly results folder.  The output file name is based upon the input file name.

    Creating a Template Script

    As you can see, this job follows a general template.  There are three main parameters that change in each job:

    1. The week 
      • Used as part of the job name
      • Used to find the proper data file to copy to the nodes local disk
      • Used to copy the results to the correct folder
    2. The data source
      • Used to find the proper data file to copy to the nodes local disk
    3. The data file's name
      • Used as part of the job name
      • Used to find the proper data file to copy to the nodes local disk
      • Used to specify both the input and output file to the program full_analysis
      • Used to copy the results to the correct folder

    If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

    #PBS -l nodes=1:ppn=12
    #PBS -l walltime=80:00:00
    #PBS -n ${WEEK}_${DATA}
    
    # Copy input data to the nodes fast local disk 
    cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR
    cd $TMPDIR
    
    # Run the analysis 
    full_analysis ${DATA}.in ${DATA}.out
    
    # Copy results to proper folder
    cp  ${DATA}.out ~/${WEEK}/results
    

    Automating Job Submission

    We can now use qsub's -v option to pass parameters to our template script.  The format for passing parameters is:

    qsub -v par_name=par_value[,par_name=par_value...] script.sh

    Submitting a 100 jobs using qsub -v option manually does not make our task much easier than modifying and submitting each job one by one.  To complete our task we need to automate the submission of our jobs.  We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

    Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.

    Here is the script that submits the jobs using the parameters:

    #!/usr/bin/env python
    import csv, subprocess
    
    parameter_file_full_path = "/nfs/12/user0123/week42/job_params.csv"
    
    with open(parameter_file_full_path, "rb") as csvfile:
        reader = csv.reader(csvfile)
        for job in reader:
            qsub_command = """qsub -v WEEK={0},SOURCE={1},DATA={2} template_1.pbs""".format(*job)
    
            #print qsub_command # Uncomment this line when testing to view the qsub command
    
            # Comment the following 3 lines when testing to prevent jobs from being submitted
            exit_status = subprocess.call(qsub_command, shell=True)
            if exit_status is 1:  # Check to make sure the job submitted
                print "Job {0} failed to submit".format(qsub_command)
    print "Done submitting jobs!"

    This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values.  If the qsub command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display.  The jobs will be submitted using the general format:

    qsub -v WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.pbs

    Where WEEK_VALUE ,SOURCE_VALUE, and DATA_VALUE are the first, second, and third comma separated values in the CSV file's current row, and template_1.pbs is the name of our template script.

    Creating a CSV File

    We now need to create a CSV file with parameters for each job.  This can be done with a regular text editor or using a spreadsheet editor such as excel.  By default you should use commas as your delimiter.  

    Here is our CSV file with parameters:

    week42,source1,data1
    week42,source1,data2
    week42,source1,data3
    ...
    week42,source2,data98
    week42,source2,data99
    week42,source2,data100
    

    The submit script would read in the first row of this CSV file and form and  submit the following qsub command:

    qsub -v WEEK=week42,SOURCE=source1,DATA=data1 template_1.pbs

    Submitting Jobs

    Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.  

    Before submitting a large amount of jobs for the first time using this method it is recommended you test our your implimentation with a small amount of jobs.

    HOWTO: Transfer files using Globus Connect

    Globus Connect Logo

    Globus Connect is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems.  It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

    Globus can be used for both file transfers between OSC and:

    • A computing institution with Globus installed (check with your site provider for availability) or
    • A personal computer (known as a personal endpoint)

    Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip to Usage.

    More on how Globus works can be found on the Globus "How It Works" page.

    If you are looking to transfer smaller sized files you can utilize OnDemand's file transfer capabilities, or use a SFTP client to connect to  sftp.osc.edu . Our general recommendation is that for small files - measured in MB to several hundred MB - to use OnDemand or SFTP. You can continue to use SFTP and get reasonable performance up to file sizes of several GB. For transfers of several GB or larger, you should consider using Globus Online.

    Install Globus Connect Personal

    To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

    1. Sign up for a free Globus account
    2. Download the Globus Connect Personal Client 
      • Click "Manage Endpoints" under the "Manage Transfers/Data" menu
      • Click "add Globus Connect" on the top-right of the page
      • Choose a unique name for your endpoint and generate the setup key
      • Download the Globus Connect client for your operating system
    3. Install Globus Connect Personal Client
      • Windows
        1. Run the Installer
        2. Copy-Paste the setup key to complete the installation
      • Mac​​
        1. Mount your drives
        2. Copy the Globus Client to your application Folder
        3. Start The Globus Client, and enter the provided setup key
      • Linux
        1. Un-tar the .tgz file with the command tar -zxvf
        2. Run globusconnect , found within the unzipped directory
        3. Copy-Paste the setup key when prompted
    4. (Optional) Changing directories accessible to Globus

    By default Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

    Windows

    1. Start Globus Connect Personal
    2. Go to "Options"
    • Add directories/files using the  "+" button
    • Remove directories/files using the "-" button
    • Revert to the default accessible directories/files using the "Reset to Defaults" button
    • Any changes you make are not made permanent until you press the "Save" button

    ​​Mac

    1. Start Globus Connect Personal
    2. Go to Preferences -> Access
    • Add directories/files using the  "+" button
    • Remove directories/files using the "-" button
    • Revert to the default accessible directories/files using the "Reset to Defaults" button
    • Any changes you make are not made permanent until you press the "Save" button

    ​Linux

    1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma separated fields as below
      <path1>,<sharing flag>,<R/W flag>
      <path2>,<sharing flag>,<R/W flag>
      <path3>,<sharing flag>,<R/W flag>
      ...
      

      Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

      Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

      R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

    2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below
    $ ./globusconnectpersonal -stop
    $ ./globusconnectpersonal -start &
    

    Usage

    1. Login to Globus and navigate to the "Transfer Files" under the "Manage Data" menu
    2. Enter your end point in one of the boxes
      • If transferring to a computer with Globus Connect Personal installed, this will be the unique name chosen during installation. It is also listed under "Administered by Me"
    3. Enter osc#gcs (OSC Globus Connect Server) as the other endpoint
      • Enter your OSC username and password for the authetication
    4. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transfer
    If you are doing a large transfer you should transfer to/from the parallel file system or project space for best performance.
    Once a transfer has begun, you do not need to keep the Globus webpage up, but you will need to make sure the Globus Connect Personal Client is running on your computer until it has completed.  If the transfer is interrupted for any reason, Globus will attempt to re-initiate the transfer automatically.

    Further Reading

    HOWTO: Add InCommon Authentication to Globus

    (OPTIONAL) Adding InCommon Authentication 

    Adding InCommon authentication to your Globus account allows you to login to Globus Online using your university credentials.  Using this process you can store your Globus username password for safe keeping, and instead use your university username and password to login.  If your already logged in to your university authentication system, logging in to Globus can be as simple as two clicks away.

    To use this feature, your university needs to be a InCommon participant.  Some Ohio universities active in InCommon include: Ohio State University, Case Western University, Columbus State Community College, Miami University, Ohio Northern University, Ohio University, University of Findlay, University of Dayton, and many more.  

    For a complete list, visit https://incommon.org/participants/ .

    To add InCommon Authentication:
    1. Login to Globus Online
    2. Go to "Manage Identities" under your username
    3. Click "Add External Identity"
      • Choose a name for the Identity Settings. 
      • Choose InCommon / CILogon from the drop down menu
    4. On the next page, choose your University / Identity Provider
    • Click "Remember this selection"
    • Click "Log on"
    • You may be prompted to login to your university authentication system if you are not already

    When you go to login next, click "alternative login" and then "InCommon / CILogon".  Select your university on the next page, and login using your university credentials.  Globus will remember this preference, and automatically prompt you to login using your university authentication next time.

    HOWTO: Use Docker and Singularity Containers at OSC

    It is now possible to run Docker and Singularity containers on the Owens cluster at OSC. Single-node jobs are currently supported, including GPU jobs; MPI jobs are planned for the future.

    From the Docker website:  "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

    This document will describe how to run Docker and Singularity containers on Owens. You can use containers from Docker Hub, Singularity Hub, or any other source. As examples we will use vsoch/singularity-hello-world and ubuntu from Docker Hub.

    If you want to create your own container or modify a container, you will have to do it on a system where you have root access. You cannot build or update a container on Owens or any other OSC system.

    Getting help

    The most up-to-date help on Singularity comes from the command itself.

    singularity help
    
    singularity help exec
    

    The Singularity web site has documentation including a User Guide and examples.

    Setting up your environment for Singularity usage

    To load the Singularity environment on Owens, run the following command:

    module load singularity
    

    Accessing a container

    A Singularity container is a single file with a .img or .simg extension.

    You can either download (“pull”) a container from a hub or run it directly from the hub. Or just copy the container from anywhere.

    Examples of “pulling” a container:

    Example:  Pull from the 7.2.0 branch of the gcc repository on Docker Hub. The 7.2.0 is called a tag.

    singularity pull docker://gcc:7.2.0
    

    Filename:  gcc-7.2.0.img

    Example:  Pull the vsoch/singularity-hello-world container from the Singularity hub. Since no tag is specified it pulls from the master branch of the repository.

    singularity pull shub://vsoch/hello-world
    

    Filename:  vsoch-hello-world-master.img

    Example:  Pull an Ubuntu container from Docker Hub.

    singularity pull docker://ubuntu
    

    Filename:  ubuntu.img

    Running a container

    There are four ways to run a container under Singularity.

    You can do all this either in a batch job or on a login node. (Don’t run on a login node if the container will be performing heavy computation, of course.)

    We note that the operating system on Owens is Red Hat:

    [owens-login01]$ cat /etc/os-release
    NAME="Red Hat Enterprise Linux Server"
    VERSION="7.3 (Maipo)"
    ID="rhel"
    ID_LIKE="fedora"
    VERSION_ID="7.3"
    PRETTY_NAME="Red Hat Enterprise Linux"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
    HOME_URL="https://www.redhat.com/"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"
    REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
    REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
    REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
    REDHAT_SUPPORT_PRODUCT_VERSION="7.3"
    

    In the examples below we will often check the operating system to show that we are really inside a container.

    Run container like a native command

    If you simply run the container image it will execute the container’s runscript.

    Example:  Run an Ubuntu shell. (See also the “shell” sub-command below.)

    [owens-login01]$ ./ubuntu.img
    [owens-login01]$ cat /etc/os-release
    NAME="Ubuntu"
    VERSION="16.04.3 LTS (Xenial Xerus)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 16.04.3 LTS"
    VERSION_ID="16.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    VERSION_CODENAME=xenial
    UBUNTU_CODENAME=xenial
    [owens-login01]$ exit
    exit
    [owens-login01]$
    

    Example:  Run vsoch/hello-world

    Note that this container returns you to your native OS after you run it.

    [owens-login01]$ ./vsoch-hello-world-master.img
    RaawwWWWWWRRRR!!
    [owens-login01]$
    

    Use the “run” sub-command

    The Singularity “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.

    Example:  Run a container from a local file

    [owens-login01]$ singularity run vsoch-hello-world-master.img
    RaawwWWWWWRRRR!!
    [owens-login01]$
    

    Example:  Run a container from a hub without explicitly downloading it

    [owens-login01]$ singularity run shub://vsoch/hello-world
    Progress |===================================| 100.0%
    RaawwWWWWWRRRR!!
    [owens-login01]$
    

    Use the “exec” sub-command

    The Singularity “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.

    Example:  Find out what operating system the vsoch/hello-world container uses

    [owens-login01]$ singularity exec vsoch-hello-world-master.img cat /etc/os-release
    NAME="Ubuntu"
    VERSION="14.04.5 LTS, Trusty Tahr"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 14.04.5 LTS"
    VERSION_ID="14.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    [owens-login01]$ 
    

    Use the “shell” sub-command

    The Singularity “shell” sub-command invokes an interactive shell within a container.

    Example:  Examine the contents of the vsoch/hello-world container

    Note:  A container’s runscript is the file /singularity within the container.

    [owens-login01]$ singularity shell vsoch-hello-world-master.img
    Singularity: Invoking an interactive shell within container...
    Singularity vsoch-hello-world-master.img:~/singularity> ls /
    bin   dev  home  lib64       media  opt   rawr.sh  run   singularity  sys  users  var
    boot  etc  lib   lost+found  mnt    proc  root     sbin  srv          tmp  usr
    Singularity vsoch-hello-world-master.img:~/singularity> cat /singularity
    #!/bin/sh
    exec /bin/bash /rawr.sh
    Singularity vsoch-hello-world-master.img:~/singularity> cat /rawr.sh
    #!/bin/bash
    echo "RaawwWWWWWRRRR!!"
    Singularity vsoch-hello-world-master.img:~/singularity> exit
    exit
    [owens-login01]$
    

    Example:  Run an Ubuntu shell. Note the “Singularity” prompt within the shell.

    [owens-login01]$ singularity shell ubuntu.img
    Singularity: Invoking an interactive shell within container...
    Singularity ubuntu.img:~/singularity> cat /singularity
    #!/bin/sh
    exec /bin/bash "$@"Singularity ubuntu.img:~/singularity> exit
    exit
    [owens-login01]$
    

    File system access

    When you use a container you run within the container’s environment.  The directories available to you by default from the host environment (Owens) are

    • your home directory
    • working directory (directory you were in when you ran the container)
    • /fs/project
    • /fs/scratch
    • /tmp

    If you run the container within a job you will have the usual access to the $PFSDIR environment variable. In order to access $TMPDIR you must specify it on the command line as in this example:

    SINGULARITYENV_TMPDIR=$TMPDIR singularity shell ubuntu.img
    
    Note: Most environment variables are passed into the container by default, but for obscure reasons $TMPDIR is not. The above syntax passes it in explicitly.

    GPU usage within a container

    If you have a GPU-enabled container you can easily run it on Owens just by adding the --nv flag to the singularity exec or run command.  The example below comes from the exec command section of Singularity User Guide.  It runs a TensorFlow example using a GPU on Owens.  (Output has been omitted from the example for brevity.)

    [owens-login01]$ qsub -I -l nodes=1:ppn=28:gpus=1
    ...
    [o0756]$ cd $PBS_O_WORKDIR
    [o0756]$ module use /usr/local/share/lmodfiles/project/osc
    [o0756]$ module load singularity
    [o0756]$
    
     git clone https://github.com/tensorflow/models.git
    
    [o0756]$
    
     singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
    python ./models/tutorials/image/mnist/convolutional.py
    
    

    References

     

    HOWTO: Use NFSv4 ACL

    This document shows you how to use the NFSv4 ACL permissions system. An ACL is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certian file or directory by user or group. NFSv4 ACLs provide more specific options than typical POSIX read/write/execute permissions used in most systems.

    Understanding NFSv4 ACL

    This is an example of an NFSv4 ACL

    A::user@nfsdomain.org:rxtncy
    
    A::alice@nfsdomain.org:rxtncy
    
    A::alice@nfsdomain.org:rxtncy
    
    A::alice@nfsdomain.org:rxtncy
    

    The following sections will break down this example from left to right and provide more usage options

     

    ACE Type

    The 'A' in the example is known as the ace type. The 'A' denotes "Allow" meaning this ACL is allowing the user or group to perform actions requiring permissions. Anything that is not explicitly allowed is denied by default.

    Note: 'D' can denote a Deny ACE. While this is a valid option, this ACE type is not reccomended since any permission that is not explicity granted is automatically denied meaning Deny ACE's can be redundant and complicated.

     

    ACE Flags

    The above example could have a distinction known as a flag shown below

    A:d:user@osc.edu:rxtncy
    

    The 'd' used above is called an inheritence flag. This makes it so the ACL set on this directory will be automatically established on any new subdirectories. Inheritence flags only work on directories and not files. Multiple inheritence flags can be used in combonation or omitted entirely. Examples of inheritence flags are listed below:

    Flag Name Function
    d directory-inherit New subdirectories will have the same ACE
    f file-inherit New files will have the same ACE minus the inheritence flags 
    n no-propogate inherit New subdirectories will inherit the ACE minus the inheritence flags
    i inherit-only New files and subdirectories will have this ACE but the ACE for the directory with the flag is null

     

    ACE Principal

    The 'user@nfsdomain.org' is a principal. The principle denotes the people the ACL is allowing access to. Principals can be the following:

    • A named user
    • Special principals
      • OWNER@
      • GROUP@
      • EVERYONE@
    • A group
      • Note: When the principal is a group, you need to add a group flag, 'g', as shown in the below example
      • A:g:group@osc.edu:rxtncy
        

     

    ACE Permissions

    The 'rxtncy' are the permissions the ACE is allowing. Permissions can be used in combonation with each other. A list of permissions and what they do can be found below:

    Permission Function
    r read-data (files) / list-directory (directories)
    w write-data (files) / create-file (directories)
    a append-data (files) / create-subdirectory (directories)
    x execute (files) / change-directory (directories
    d delete the file/directory
    D delete-child : remove a file or subdirectory from the given directory (directories only)
    t read the attributes of the file/directory
    T write the attribute of the file/directory
    n read the named attributes of the file/directory
    N write the named attributes of the file/directory
    c read the file/directory ACL
    C write the file/directory ACL
    o change ownership of the file/directory

     

    Note: Aliases such as 'R', 'W', and 'X' can be used as permissions. These work simlarly to POSIX Read/Write/Execute. More detail can be found below.

    Alias Name Expansion
    R Read rntcy
    W Write watTNcCy (with D added to directory ACE's
    X Execute xtcy

     

    Using NFSv4 ACL

    This section will show you how to set, modify, and view ACLs

     

    Set and Modify ACLs

    To set an ACE use this command:

    nfs4_setfacl [OPTIONS] COMMAND file
    

    To modify an ACE, use this command:

    nfs4_editfacl [OPTIONS] file
    

    Where file is the name of your file or directory. More information on Options and Commands can be found below.

     

    Options

    Options can be used in combination or ommitted entirely. A list of options is shown below:

    Option Name Function
    -R recursive Applies ACE to a directory's files and subdirectories
    -L logical Used with -R, follows symbolic links
    -P physical Used with -R, skips symbolic links

     

    Commands

    Commands are only used when first setting an ACE. Commands and their uses are listed below.

     

    View ACLs

    To view ACLs, use the following command:

    nfs4_getfacl file
    

    Where file is your file or directory

     

    Service: 

    HOWTO: Use VNC in a batch job

    SSHing directly to a compute node at OSC - even if that node has been assigned to you in a current batch job - and starting VNC is an "unsafe" thing to do. When your batch job ends (and the node is assigned to other users), stray processes will be left behind and negatively impact other users. However, it is possible to use VNC on compute nodes safely.

    You can use OnDemand, which is a much easier way to access desktops. If your work is not a very large, very intensive computation (for example, you do not expect to saturate all of the cores on a machine for a significant portion of the time you have the application you require open - e.g., you are using the GUI to set up a problem for a longer non-interactive compute job), you can choose one VDI under "Virtual Desktop Interface" from "Desktops" menu. Otherwise, please use "Interactive HPC" from Desktops" menu.

    The examples below are for Oakley. If you use other systems, please see this page for supported versions of TurboVNC on our systems.

    Starting your VNC server

    Step one is to create your VNC server inside a batch job.

    Option 1: Interactive

    The preferred method is to start an interactive job, requesting an entire node, and then once your job starts, you can start the VNC server.

    qsub -I -l nodes=1:ppn=12:gpus=2:vis
    

    This command requests an entire GPU node, and tells the batch system you wish to use the GPUs for visualization. This will ensure that the X11 server can access the GPU for acceleration. In this example, I have not specified a duration, which will then default to 1 hour.

    module load virtualgl
    module load turbovnc/2.0
    

    Then start your VNC server. (The first time you run this command, it may ask you for a password - this is to secure your VNC session from unauthorized connections. Set it to whatever password you desire. We recommend a strong password.)

    vncserver
    

    The output of this command is important: it tells you where to point your client to access your desktop. Specifically, we need both the host name (before the :), and the screen (after the :).

    New 'X' desktop is n0302.ten.osc.edu:1
    

    Option 2: Batch

    This option is less optimal, because it is slightly more difficult to get the hostname and screen. However, by submitting a non-interactive batch job, you can go away and have the system email you when your desktop is ready to be connected to, and more importantly if your SSH connection to OSC is somewhat unstable and intermittent, you do not run the risk of being disconnected during your interactive session and having your VNC server terminated. In general, it is recommended you only use this option if running via an interactive session is not feasible.

    In order to start an VNC session non-interactively, you can submit the following script to the scheduler using qsub (adjusting your walltime to what you need):

    #PBS -l nodes=1:ppn=12:gpus=2:vis
    #PBS -l walltime=00:15:00
    #PBS -m b
    #PBS -N VNCjob
    #PBS -j oe
    module load virtualgl
    module load turbovnc/2.0
    vncserver
    sleep 100
    vncpid=`pgrep -s 0 Xvnc`
    while [ -e /proc/$vncpid ]; do sleep 0.1; done
    

    This script will send you an email when your job has started, which includes the hostname.

    PBS Job Id: 9200155.oak-batch.osc.edu
    Job Name:   VNCjob
    Exec host:  n0311/0-11
    Begun execution
    

    The screen is virtually always "1", unless someone else started a VNC server on that node outside of the batch system. You can verify the output of the vncserver command by using qpeek on a login node:

    qpeek jobid
    

    Where "jobid" is the batch system job number, for example, "9200155".

     

    Connecting to your VNC server

    Because the compute nodes of our clusters are not directly accessible, you must log in to one of the login nodes and allow your VNC client to "tunnel" through SSH to the compute node. The specific method of doing so may vary depending on your client software.

    Linux/MacOS

    Option 1: Manually create an SSH tunnel 

    I will be providing the basic command line syntax, which works on Linux and MacOS. You would issue this in a new terminal window on your local machine, creating a new connection to Oakley.

    ssh -L 5901:n0302.ten.osc.edu:5901 guilfoos@oakley.osc.edu
    

    Open your VNC client, and connect to "localhost:1" - this will tunnel to the correct node on Oakley.

    Option 2: Use your VNC software to tunnel 

    This example uses Chicken of the VNC, a MacOS VNC client.

    The default window that comes up for Chicken requires the host to connect to, the screen (or port) number, and optionally allows you to specify a host to tunnel through via SSH. This screenshot shows a proper configuration for the output of vncserver shown above. Substitute your host, screen, and username as appropriate.

    When you click [Connect], you will be prompted for your HPC password (to establish the tunnel, provided you did not input it into the "password" box on this dialog), and then (if you set one), for your VNC password. If your passwords are correct, the desktop will display in your client.

     

    Windows

    This example shows how to create a SSH tunnel through your ssh client.  We will be using Putty in this example, but these steps are applicable to most SSH clients.

    First, make sure you have x11 forwarding enabled in your SSH client.

    Next, open up the port forwarding/tunnels settings and enter the hostname and port you got earlier in the destination field.  You will need to add 5900 to the port number when specifiying it here.  Some clients may have separate boxes for the desination hostname and port.  

    For source port, pick a number between 11-99 and add 5900 to it.  This number between 11-99 will be the port you connect to in your VNC client.

    Make sure to add the forwaded port, and save the changes you've made before exiting the configutations window.

    PuTTY Tunnel Configuration Settings

    Now start a SSH session to the respective cluster your vncserver is running on.  The port forwarding will automatically happen in the background.  Closing this SSH session will close the forwarded port; leave the session open as long as you want to use VNC.

    Now start a VNC client.  TurboVNC has been tested with our systems and is recommended.  Enter localhost:[port], replacing [port] with the port between 11-99 you chose earlier.

    New TurboVNC Connection

    If you've set up a VNC password you will be prompted for it now.  A desktop display should pop up now if everything is configured correctly.

    How to Kill a VNC session?

    Occasionally you may make a mistake and start a VNC server on a login node or somewhere else you did not want to.  In this case it is important to know how to properly kill your VNC server so no processes are left behind.

    The command syntax to kill a VNC session is:

    vncserver -kill :[screen]
    

    In the example above, screen would be 1.

    You need to make sure you are on the same node you spawned the VNC server on when running this command.

    Service: 
    Fields of Science: 

    HOWTO: Use an Externally Hosted License

    Many software packages require a license.  These licenses are usually made available via a license server, which allows software to check out necessary licenses.  In this document external refers to a license server that is not hosted inside OSC.

    If you have such a software license server set up using a license manager, such as FlexNet, this guide will instruct you on the necessary steps to connect to and use the licenses at OSC.

    Users who wish to host their software licenses inside OSC should consult OSC Help.

     

    You are responsible for ensuring you are following your software license terms.  Please ensure your terms allow you to use the license at OSC before beginning this process!

    Introduction

    Broadly speaking, there are two different ways in which the external license server's network may be configured.  These differ by whether the license server is directly externally reachable or if it sits behind a private internal network with a port forwarding firewall.  

    If your license server sits behind a private internal network with a port forwarding firewall you will need to take additional steps to allow the connection from our systems to the license server to be properly routed. 

    License Server is Directly Externally Reachable

    Diagram showing network setup of license server that is directly externally reachable

     

    License Server is Behind Port Forwarding Firewall

    Diagram showing network setup of license server that is behind a port forwarding firewall

     

    Unsure?

    If you are unsure about which category your situation falls under contact your local IT administrator.

    Configure Remote Firewall

    In order for connections from OSC to reach the license server, the license server's firewall will need to be configured.  All outbound network traffic from all of OSC's compute nodes are routed through a network address translation host (NAT), or two backup servers.

    The license server should be configured to allow connections from the following IP addresses to the SERVER:PORT where the license server is running:

    • nat.osc.edu (192.157.5.13)
    • 192.148.248.35
    • 192.148.248.186

    A typical FlexNet-based license server uses two ports: one is server port and the other is demon port, and the firewall should be configured for the both ports. A typical license file looks, for example,

    SERVER licXXX.osc.edu 0050XXXXX5C 28000
    VENDOR {license name} port=28001
    

    In this example, "28000" is the server port, and "28001" is the demon port. The demon port is not mandatory if you use it on a local network, however it becomes necessary if you want to use it outside of your local network. So, please make sure you declared the demon port in the license file and configured the firewall for the port.

    Confirm Configuration

    The firewall settings should be verified by attempting to connect to the license server from the compute environment using telenet.

    Get on to a compute node by requesting a short, small, interactive job and test the connection using telenet:

    telnet <License Server IP Address> <Port#>
    

    (Recommended) Restrict Access to IPs/Usernames

    It is also recommended to restrict accessibility using the remote license server's access control mechanisms, such as limiting access to particular usernames in the options.dat file used with FlexNet-based license servers.

    For FlexNet tools, you can add the following line to your options.dat file, one for each user.

    INCLUDEALL USER <OSC username>
    

    If you have a large number of users to give access to you may want to define a group using GROUP within the options.dat file and give access to that whole group using INCLUDEALL GROUP <group name> .

    Users who use other license managers should consult the license manager's documentation.

    Modify Job Environment to Point at License Server

    The software must now be told to contact the license server for it's licenses.  The exact method of doing so can vary between each software package, but most use an environment variable that specifies the license server IP address and port number to use.

    For example LS DYNA uses the environment variable LSTC_LICENSE and LSTC_LICENSE_SERVER to know where to look for the license.  The following lines would be added to a job script to tell LS-DYNA to use licenses from port 2345 on server 1.2.3.4:

    setenv LSTC_LICENSE network
    setenv LSTC_LICENSE_SERVER 2345@1.2.3.4
    

    License Server is Behind Port Forwarding Firewall

    If the license server is behind a port forwarding firewall, and has a different IP address from the IP address of the firewall, additional steps must be taken to allow connections to be properly routed within the license server's internal network.  

    1. Use the license server's fully qualified domain name in the environment variables instead of the IP address.
    2. Contact OSC Help to have the firewall IP address mapped to the fully qualified domain name.

     

    Software Specific Details

    The following outlines details particular to a specific software package.  

    ANSYS

    Uses the following environment variables:

    
    ANSYSLI_SERVERS=<port>@<IP>
    ANSYSLMD_LICENSE=<port>@<IP>
    

     

    HOWTO: Use ulimit command to set soft limits

    This document shows you how to set soft limits using ulimit command.

    The ulimit command sets or reports user process resource limits. The default limits are defined and applied when a new user is added to the system. Limits are categorized as either soft or hard. With the ulimit command, you can change your soft limits for the current shell environment, up to the maximum set by the hard limits. You must have root user authority to change resource hard limits.

    Syntax

    ulimit [-H] [-S] [-a] [-c] [-d] [-f] [-m] [-n] [-s] [-t] [Limit]
    flags description

    -a

    Lists all of the current resource limits

    -c

    Specifies the size of core dumps, in number of 512-byte blocks

    -d

    Specifies the size of the data area, in number of K bytes

    -f

    Sets the file size limit in blocks when the Limit parameter is used, or reports the file size limit if no parameter is specified. The -f flag is the default

    -H

    Specifies that the hard limit for the given resource is set. If you have root user authority, you can increase the hard limit. Anyone can decrease it

    -m

    Specifies the size of physical memory, in number of K bytes

    -n

    Specifies the limit on the number of file descriptors a process may have

    -s

    Specifies the stack size, in number of K bytes

    -S

    Specifies that the soft limit for the given resource is set. A soft limit can be increased up to the value of the hard limit. If neither the -H nor -S flags are specified, the limit applies to both

    -t

    Specifies the number of seconds to be used by each process

    The limit for a specified resource is set when the Limit parameter is specified. The value of the Limit parameter can be a number in the unit specified with each resource, or the value "unlimited". For example, to set the file size limit to 51,200 bytes, use:

    ulimit -f 100

    To set the size of core dumps to unlimited, use:

    ulimit –c unlimited

    How to change ulimit for a MPI program

    ulimit command affects the current shell environment. When a MPI program is started, it does not spawn in the current shell. You have to use mpiexec to start a wrapper script that sets the limit if you want to set the limit for each process. Below is how you set the limit for each shell (We use  ulimit –c unlimited to allow unlimited core dumps, as an example): 

    1. Prepare your batch job script named "myjob" as below (Here, we request 5-hour 2-node on Oakley cluster):
    #PBS -l nodes=2:ppn=12
    #PBS -l walltime=5:00:00
    #PBS ...
    
    #!/bin/bash
    
    cd $PBS_O_WORKDIR
    ...
    mpiexec ./test1
    ...
    1. Prepare the wrapper script named "test1" as below:
    #!/bin/bash
    ulimit –c unlimited
    .....(your own program)
    1. qsub myjob
    Service: 

    HOWTO: submit homework to repository at OSC

    This page outlines a way a professor can set up a file submission system at OSC for his/her classroom project.

    Usage for Professor

    After connect to OSC system, professor runs submit_prepare as

    $ /fs/scratch/xwang/bin/submit_prepare

    Follow the instructions. It will create designated directory where students submit their assignments, as well as generate submit for students used to submit homework to OSC, both of which are located in the directory specified by the professor.

    Note:

    The PI can also enforce deadline by simply changing the permission of the submission directory or renaming the submission directory at the deadline.

    (Only works on Owens): One way is to use at command following the steps below:

    • Use at command to specify the deadline:
    at [TIME]

    where TIME is formatted HH:MM AM/PM MM/DD/YY. For example:

    at 2:30 PM 08/21/2017
    • After running this command, run:
    $ chmod 700 [DIRECTORY]

    where DIRECTORY is the assignment folder to be closed off.

    • Enter [ctrl+D] to submit this command.

    The permission of DIRECTORY will be changed to 700 at 2:30PM, August 21, 2018. After that, student will get an error message when he/she tries to submit assignment to this directory.

    Usage for Students

    Student should create one directory which includes all the files he/she wants to submit before running this script to sumit his/her assignment. Also, the previous submission of the same assignment from the student will be replaced by the new submission. 

    To submit the assignemt, student runs submit after connecting to OSC system as

    $ /path/to/directory/from/professor/submit

    Follow the instructions. It will allow students to submit assignment to designated directory speficifed by professor and send a confirmation email, or return error message.

    Service: 

    Citation

    The Ohio Supercomputer Center provides High Performance Computing resources and expertise to academic researchers across the State of Ohio. Any paper citing this document has utilized OSC to conduct research on our production services. OSC is a member of the Ohio Technology Consortium, a division of the Ohio Department of Higher Education.

    Hardware

    OSC services can be cited by visiting the documentation for the service in question and finding the "Citation" page (located in the menu on the left of the screen).

    HPC systems currently in production use can be found here: https://www.osc.edu/supercomputing/hpc

    Decommissioned HPC systems can be found here: https://www.osc.edu/supercomputing/hpc/decommissioned.

    Logos

    Please refer to our branding webpage

    Citing OSC

    We prefer that you cite OSC when using our services, using the following information, taking into account the appropriate citation style guidelines. For your convenience, we have included the citation information in BibTeX and EndNote formats.

    Ohio Supercomputer Center. 1987. Ohio Supercomputer Center. Columbus OH: Ohio Supercomputer Center. http://osc.edu/ark:/19495/f5s1ph73.

    BibTeX:

    @misc{OhioSupercomputerCenter1987,
    ark = {ark:/19495/f5s1ph73},
    howpublished = {\url{http://osc.edu/ark:/19495/f5s1ph73}},
    year  = {1987},
    author = {Ohio Supercomputer Center},
    title = {Ohio Supercomputer Center}
    }
    

    EndNote:

    %0 Generic
    %T Ohio Supercomputer Center
    %A Ohio Supercomputer Center
    %R ark:/19495/f5s1ph73
    %U http://osc.edu/ark:/19495/f5s1ph73
    %D 1987

    Here is an .ris file to better suit your needs. Please change the import option to .ris

    Documentation Attachment: 

    OSCusage

    Introduction

    OSCusage is command developed at OSC for use on OSC's systems.  It allows for a user to see information on their project's current RU balance, including which users and jobs incurred what charges.

    Availability

    Oakley ruby owens
    X X X

     

    Usage

    OSCusage takes the following options and parameters.

    usage: OSCusage [-qvn] -[r|h] [MM/DD/YYYY] [MM/DD/YYYY]
             or [-i]
    Note: the year requires all 4 digits.
    
    OSCusage can only provide information on a user's primary project, visible with the command id -gn.  If you are a part of other projects, visible under id -Gn, you will be unable to pull information on those projects.  Contact OSC Help for assistance in getting such information.

    Today's Usage

    Running OSCusage with no options or parameters specified will provide RU usage information per a user for the current day.

    $ OSCusage
    
                         Usage Statistics for project PAS0435
                                for 5/3/2016 to 5/3/2016
    
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
            Username  Dates                            RUs  Status
    
              osu525    5/3/2016 to   5/3/2016     0.00000  ACTIVE
             *osu391    5/3/2016 to   5/3/2016     0.00000  ACTIVE
             osu6396    5/3/2016 to   5/3/2016     0.00000  ACTIVE
             osu6445    5/3/2016 to   5/3/2016     0.00000  PASSWORD
             osu8968    5/3/2016 to   5/3/2016     0.00000  ACTIVE
     ===============   PAS0435 TOTAL    0.00000

    Users on Project

    Specifying -i provides information on the accounts under the project.

    $ OSCusage -i
                            Information for project PAS0435
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
                Username    Status  Full Name
    
                 *osu391    ACTIVE  Dr. Judith Gardiner
                 osu6396    ACTIVE  Dr. Judith Gardiner
                 osu8968    ACTIVE  Koh Schooley
    
    

    RU Usage in Timeframe

    If you specify a timeframe you can get RU utilization information specifically for jobs that completed within that period.

    $ OSCusage 01/01/2016 05/01/2016
    
                         Usage Statistics for project PAS0435
                              for 01/01/2016 to 05/01/2016
    
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
            Username  Dates                            RUs  Status
    
              osu525  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             *osu391  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             osu6396  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             osu6445  01/01/2016 to 05/01/2016     0.00000  PASSWORD
             osu8968  01/01/2016 to 05/01/2016     0.15752  ACTIVE
     ===============   PAS0435 TOTAL    0.15752

    Show only my usage

    Specify -q to show only the current user's usage.

    $ OSCusage -q 01/01/2016 05/01/2016
    
                         Usage Statistics for project PAS0435
                              for 01/01/2016 to 05/01/2016
    
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
            Username  Dates                            RUs  Status
    
             osu8968  01/01/2016 to 05/01/2016     0.15752  ACTIVE
     ===============   PAS0435 TOTAL    0.15752
    

    Detailed Charges Breakdown

    Specify -v to get detailed information on what job's incurred what charges.

    $ OSCusage -v 01/01/2016 05/01/2016
    
                         Usage Statistics for project PAS0435
                              for 01/01/2016 to 05/01/2016
    
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
                            Start                 RUs     Charge           Status
            Username  Date  Date & Time          Used       Type     Queue           Job
    
     -------------            osu525 TOTAL    0.00000
     -------------           *osu391 TOTAL    0.00000
     -------------           osu6396 TOTAL    0.00000
     -------------           osu6445 TOTAL    0.00000
             osu8968 03/30  03/29 15:14:31    0.00277   CPU-Ruby    serial  E      STDIN
             osu8968 03/22  03/21 11:38:14    0.00005 CPU-Oakley    serial  E   login_en
             osu8968 03/22  03/21 11:48:25    0.00005 CPU-Oakley    serial  E   test_env
             osu8968 03/10  03/09 12:30:36    0.00066 CPU-Oakley    serial  E   test_pat
             osu8968 02/20  02/19 17:14:23    0.01266 CPU-Oakley  parallel  E   initial_
             osu8968 02/20  02/19 17:16:24    0.14133 CPU-Oakley  parallel  E   initial_
     -------------           osu8968 TOTAL    0.15752
     ===============   PAS0435 TOTAL    0.15752
    
    

    View Usage in Hours

    Specify -h to view the usage information in core hours, as opposed to RUs.

    $ OSCusage -h 01/01/2016 05/01/2016
    
                         Usage Statistics for project PAS0435
                              for 01/01/2016 to 05/01/2016
    
                       PI: Dr. Judith Gardiner
                                 RU Balance: -17.88635
    
                     Note: Listing is in HOURS, as opposed to RUs.
    
            Username  Dates                            HRS  Status
    
              osu525  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             *osu391  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             osu6396  01/01/2016 to 05/01/2016     0.00000  ACTIVE
             osu6445  01/01/2016 to 05/01/2016     0.00000  PASSWORD
             osu8968  01/01/2016 to 05/01/2016     1.57520  ACTIVE
     ===============   PAS0435 TOTAL    1.57520

    Usage on Secondary Group Projects

    Oct 3, 2017: This function is broken now. We are working to fix it soon. 

    Specifying -A provides usage information for all of the secondary group projects authorized under your account, including the primary project. This is mainly for Annual Allocations PIs. 

    $ OSCusage -A
    $ OSCusage -A 01/01/2016 05/01/2016

    No examples can be shown for this command. Please contact OSC Help with questions. 

    Service: 

    Supercomputing FAQ

    General Questions

    Account Questions

    Disk Storage Questions

    Email Questions

    Linux Questions

    SSH Questions

    Batch Processing Questions

    Compiling System Questions

    Parallel Processing Questions

    Libraries/Software Questions

    Performance Analysis Questions

    Other Common Problems

    General Questions

    Who can get an account?

    Any faculty member or research scientist at an academic institution in Ohio is eligible for an academic account at OSC. These researchers/educators may request accounts for their students and collaborators. Commercial accounts are also available.  More information about applying for both academic and commercial accounts at OSC can be found at https://www.osc.edu/supercomputing/support/account

    Where should a new OSC user begin?

    The first thing you should do as a new OSC user is to check the email account that was given to OSC when you were registered.  If you are a PI, this will be the email address you entered in the primary contact information on your Project Request application. When your account is created, you will receive an email giving you information on how to start using the services at OSC.  Attached will be a letter you can keep for reference that contains your username and temporary password, as well as some other important information about using the systems at OSC.  After carefully reading the email, please first go to my.osc.edu to change your temporary password.

    Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

    Do I have to pay for supercomputer use?

    OSC receives funding from the state of Ohio. There is no charge for Ohio academic use. 

    How many supercomputers does OSC have?  Which one should I use?

    OSC currently has three HPC clusters: Owens Cluster, a 23,500+ core Dell cluster with Intel Xeon processors, Ruby Cluster, a 4800 core Intel Xeon machine from HP, and Oakley Cluster, a 8,300+ core HP Intel Xeon machine. New users have access to Owens and Oakley clusters. Ruby is currently unavailable for general access. To learn more, click here.

    How do I cite OSC in my publications?

    Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.

    How do I submit my publications and funding information to OSC?

    Please submit a list of publications, noting which cite OSC, as well as a list of current funding with your application materials for additional resources. Alternatively, please email the lists to OSC Help

    Can I receive a letter of support from OSC when I apply for outside funding?

    OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the application process for time on OSC's systems. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; number of RUs requested per year. Please allow at least two working days to process your request.

    Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware

    How do I register for a workshop?

    For a complete schedule of current training offerings, please visit the OSC Training Schedule.  To register or for more information, please email oschelp@osc.edu.  

    Where can I find documentation?

    For documentation specific to software applications, see Software.  For other available documentation, see Manuals.

    My question isn't answered here. Whom can I ask for help?

    Contact the OSC Help Desk. Support is available 24x7x365, but more complicated questions will need to wait for regular business hours (Monday - Friday, 9am - 5pm) to be addressed. More information on the OSC supercomputing help desk can be found on our Support Services page.

    Something seems to be wrong with the OSC systems.  Should I contact the help desk?

    Information will be coming soon for guidelines on reporting possible system problems.

    Where can I find logos for my presentations, posters, etc.?

    Please see our citation webpage.

    Account Questions

    What are projects and accounts?

    An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project, and authorized users can have more than one account. 

    How do I get/renew an account?

    For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

    I'm a faculty member.  How do I get accounts for my students?

    If an eligible principal investigator is new to OSC, he/she can apply for a start-up request and include the contact information for authorized users. Please refer to http://app.osc.edu/cgi-bin/app/startup for the on-line form. If an eligible principal investigator has a current project, he/she can email OSC Help to add the user. Authorized users do not have to be located in Ohio.

    I'm continuing the research of a student who graduated.  Can I use his/her account?

    Yes, but you must have the account transferred into your name. Have your PI send an email to oschelp@osc.edu

    I'm working closely with another student.  Can we share an account?

    No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. Ask your PI to request an account for you by emailing OSC Help.

    How do I change my password?

    You can change your password through the My OSC portal. Log in at http://my.osc.edu, and click the "change password" button at the bottom left corner of the HPC User Profile box. Please note that your password has certain requirements; these are specified on the "change password" portal. Please wait an hour before attempting to log in with the new password. For security purposes, please note that our password change policy requires a password change every 90 days.

    I want to use csh instead of bash.  How do I change the default shell?

    You can change your default shell through the My OSC portal. Log in at my.osc.edu, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about an hour for the changes to be applied.

    What is an RU?

    An RU is a resource unit. A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charging.

    How do I find my account balance?

    To see usage and balance information from any system, refer to the OSCusage page.

    NOTE: Accounting is updated once a day, so the account balance is for the previous day.

    How do I get more resources?

    To request more resources, the principal investigator must prepare a proposal that will be sent for peer review. After the review process is complete, the Allocations Committee of the Statewide Users Group meets to determine the number of resource units to be awarded. The committee meets on the second Thursday of even-numbered months. PIs should submit their proposals at least three weeks before the meeting date to ensure inclusion on the agenda.

    The application form is located at http://app.osc.edu/cgi-bin/app/. A complete application comprises the following items:

    • Contact information of new authorized users
    • Contact information of at least three recommended reviewers
    • Electronic version of PI’s resume
    • Publications list (publications, presentations, and articles that cite OSC)
    • Justification and performance (quantitative estimation of the number of RUs per run times; the number of runs for the estimated duration of the request; performance reports of the code to be used or explanations about performance; report of any optimization performed on the code
    • Proposal text

    How much will my account be charged for supercomputer usage?

    For details on charging algorithms, see Charging.

    Disk Storage Questions

    What is my disk quota?

    Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota.  See Available File Systems for more information.

    How can I determine the total disk space used by my account?

    Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

    Note: The quota information displayed at login is updated twice a day, so the information may not reflect the curent usage.

    You may display your home directory quota information with quota -s .

    How do I get more disk space?

    Your home directory quota cannot be increased. You should consider deleting, transferring, and/or compressing your files to reduce your usage. 

    A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to oschelp@osc.edu.

    See the Allocations and Accounts section for more information. 

    How can I find my largest directories?

    To reveal the directories in your account that are taking up the most disk space you can use the du , sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

    du . | sort -n | tail -n 10
    
    

    Why do I receive "no space left" error when writing data to my home directory?

    If you receive the error "No space left on device" when you try to write data to your home directory, it indicates the disk is full. First, check your home directory quota. Each user has 500 GB quota of storage and the quota information is shown when you login to our systems. If your disk quota is full, consider reducing your disk space usage. If your disk quota isn't full (usage less than 500GB), it is very likely that your disk is filled up with 'snapshot' files, which are invisible to users and used to track fine-grained changes to your files for recovering lost/deleted files. In this case, please contact OSC Help for further assitance. To avoid this situation in future, consider running jobs that do a lot of disk I/O in the temporary filesystem ($TMPDIR or $PFSDIR) and copy the final output back at the end of the run. See Available File Systems for more information.

    How can I use tar and gzip to aggregate and compress my files?

    The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

    tar -czvf src.tar.gz src/
    
    

    This archive can then be unpackaged using

    tar -xzvf src.tar.gz
    
    

    where the resulting directory/file structure is identical to what it was initially.

    The programs zip , bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

    Tar is taking too long.  Is there a way to compress quicker?

    If using tar with the options zcvf is taking too long you can instead use pigz  in conjunction with tar.   pigz does gzip compression while taking advantage of multiple cores.

    tar cvf - paths-to-archive | pigz > archive.tgz
    

    pigz defaults to using eight cores, but you can have it use more or less with the -p argument.

    tar cvf - paths-to-archive | pigz -n 4 > archive.tgz
    
    Due to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores.  If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.

    Note: pigz does not significantly improve decompression time.  

    Email Questions

    How do I change the email address OSC uses to contact me?

    Please update your email on my.osc.edu, or send your new contact information to oschelp@osc.edu

    I got an automated email from OSC.  Where can I get more information about it?

    See the Knowledge Base.

    Linux Questions

    What is Linux?

    Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

    How can I get started using Linux?

    See the Unix Basics tutorial for more information.  There are also many tutorials available on the web.

    SSH Questions

    What is SSH?

    Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

    How does SSH work?

    SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information. The public key is stored in ~/.ssh/id_rsa.pub and the private key is stored in  ~/.ssh/id_rsa. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files. 

    One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections. 

    In addition, on Owens the default SSH client config enables hashing of a user’s known_hosts file.  So if SSH is used on Owens the remote system’s SSH key is added to ~/.ssh/known_hosts in a hashed format which can’t be unhashed.  If the remote server’s SSH key changes, special steps must be taken to remove the SSH key entry:

    ssh-keygen -R <hostname>

    Can I connect without using an SSH client?

    The OSC OnDemand portal allows you to connect to our systems using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.  

    How can I upload or download files?

    Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

    Where can I find SSH and SFTP clients?

    There are many SSH and SFTP clients available, both commercial and free.  See Getting Connected for some suggestions.  

    How do I run a graphical application in an SSH session?

    Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

    Batch Processing Questions

    What is a batch request?

    On all OSC systems, batch processing is managed by the Portable Batch System (PBS). PBS batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.  

    How do I submit, check the status, and/or delete a batch job?

    PBS uses qsub to submit, qstat to check the status, and qdel to delete a batch request. For more information, see our Batch-Related Command Summary

    Can I be notified by email when my batch job starts or ends?

    Yes.  See the -m option in our PBS Directives Summary. If you are submitting a large number of jobs, this may not be a good idea.

    Why won't my job run?

    There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

    • Your account may be at or near the job count or processor count limit for an individual user.
    • Your group/project may be at or near the job count or processor count limit for a group.
    • The scheduler may be trying to free enough processors to run a large parallel job.
    •  Your job may need to run longer than the time left until the start of a scheduled downtime.
    • You may have requested a scarce resource or node type, either inadvertently or by design.

    See our Scheduling Policies and Limits for more information. 

    Why is my job being rejected with the error "Group: is not valid"?

    If you are a part of a annual allocations project, usually designated by the prefix PAA-, in addition to another project you must explicitly specify which project you want to charge a jobs usage to.  This can be done with the -A PBS directive.  For example if a user was a part of both projects PAA0999 and PAQ0343 and wanted to charge a job to PAA0999 they would need to add the following line to their script:

    #PBS -A PAA0999
    

    How can I retrieve files from unexpectedly terminated jobs?

    A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the trap command ( trap commands do not work in csh and tcsh shell batch scripts). In the batch script, the trap command needs to precede the command causing the TERMination. It could be placed immediately after the PBS header lines. Here is a generic form:

    trap "cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID;exit" TERM
    

    If a command in a batch script is killed for excessive memory usage (see Out-of-Memory (OOM) or Excessive Memory Usage for details) then the trap command may not be executed.  However, normal shell scripting can handle this situation:  the exit status of a command that may possibly cause an OOM can be checked and appropriate action taken.  Here is a Bourne shell example:

    bla_bla_big_memory_using_command_that_may_cause_an_OOM
    if [ $? -ne 0 ]; then
        cd $PBS_O_WORKDIR;mkdir $PBS_JOBID;cp -R $TMPDIR/* $PBS_JOBID
        exit
    fi
    

    Finally, if a node your job is running on crashes then the trap command may not be executed. It may be possible to recover your files from batch-managed directories in this case. Contact OSC Help for assistance.

    How can I delete all of my jobs on a cluster?

    To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:

    qselect -u <username> | xargs qdel
    

    How can I determine the number of cores in use by me or my group?

    To determine the number of cores (processors) in use by you account on a particular system run:

    showq -u $USER | grep "local jobs"
    

    To determine the number of cores (processors) in use by your primary project on a particular system run:

    showq -w acct=$GROUP | grep "local jobs"
    

    Genrerally, to see the number of cores in use by a particular project on a particular system run:

    showq -w acct=<project> | grep "local jobs"
    

    replacing <project> with the project ID.

    How to request GPU nodes for visualization?

    By default, we don't start an X server on gpu nodes because it impacts computational performance. Add vis in your GPU request such that the batch system uses the GPUs for visualization. For example, on Owens, it should be

    nodes=1:ppn=28:gpus=1:vis

    Compiling System Questions

    What languages are available?

    Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.

    What compiler (vendor) do you recommend?

    We have Intel, PGI, and gnu compilers available on all systems. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs. For more information, see our Compilation Guide.

    Will software built for one system run on another system?

    Most serial code built on one system will run on another system, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

    Parallel Processing Questions

    What is parallel processing?

    Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

    What parallel processing environments are available?

    On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available.  A summary of parallel environments will be coming soon.

    What is a core?

    A core is a processor. When a single chip contains multiple processors, they are called cores.

    I'm not seeing the performance I expected.  How can I be sure my code is running in parallel?

    We are currently working on a guide for this. Please contact OSC Help for assistance.

    Libraries/Software Questions

    What software applications are available?

    See the Software section for more information.

    Do you have a newer version of (name your favorite software)?

    Check the Software section to see what versions are installed. You can also check the installed modules using the module spider  or module avail  command.

    How do I get authorized to use a particular software application?

    Please contact OSC Help for assistance.

    What math routines are available?  Do you have ATLAS and LAPACK?

    See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. 

    Do you have NumPy/SciPy?

    The NumPy and SciPy modules are installed with the python software. See the Python software page.

    OSC does not have a particular software package I would like to use. How can I request it?

    Please refer to the Software Forms page. You will see a link to Request for Software Form. Download the form, complete the information, and attach the form to an e-mail to oschelp@osc.edu. The Statewide Users Group will consider the request.

    You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

    I have a software package that must be installed as root.  What should I do?

    Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

    What are modules?

    Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

    Performance Analysis Questions

    What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

    MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

    How do I find out about my code's performance?

    A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

    How can I optimize my code?

    There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

    Other Common Problems

    What does "CPU time limit exceeded" mean?

    Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

    My program or file transfer died for no reason after 20 minutes.  What happened?

    Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to gridftp01.osc.edu instead of to a login node.

    Why did my program die with a segmentation fault, address error, or signal 11?

    This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

    I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

    Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

    dos2unix myfile.txt 
    
    
    mac2unix myfile.txt  
    
    

    I copied my output file to a Windows system, but it doesn't display correctly.  How can I fix it?

    A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format:

    unix2dos myfile.txt  
    
    

    What IP ranges do I need to allow in my firewall to use OSC services?

    See our knowledge base article on the topic.

    Available Software

    Available Software icon

    OSC has a variety of software applications to support all aspects of scientific research. You can view the complete software list, which is being updated continually.

    Recent changes can be found by looking at the Changelog.

    OSC also offers licenses for some software packages to Ohio researchers via our statewide software program.

    Some packages are access-controlled due to license restrictions. You can find the forms necessary to request access.

    Available software by OSC System and Application

    Complete list of current software filterable by OSC system and use or field of study.

    Access Forms

    Some OSC software requires completion of a form before access can be granted.

    Statewide Software

    Statewide licensed software tools that will facilitate research.

    Software Refresh

    Information on software updates on OSC system.

    License Server Status

    Interruption details and status of the license servers.

    Browse Software

    License Server Status

    Interruption information and status of license servers are posted below. If you have any questions, please contact OSC Help .

    Current Issues

    • No active interruptions 

    Previous Issues

    • StarCCM academic license (license6) : outage from 12:00am EST Nov 27, 2017 Due to license renewal: license recovered 11:30am EST Nov 30, 2017. License updated
    • Ansys academic license (license5) : unstable from 12:20pm EST Nov 8, 2017 Due to license maintenance: stable from 1:30pm EST Nov 8, 2017. license updated
    • intel academic license (license5) : unstable from 4:00pm EST Oct 13, 2017 Due to license update: stable from 4:30pm EST Oct 13, 2017. license updated
    • Starccm academic license (license6) : expecting outage from Sep 23, 2017 Due to license renewal: license recovered 08:00PM EST Sep 25, 2017
    • matlab academic license (license5) : unstable from 11:40am EST Jul 19, 2017 Due to license update: stable from 12:07pm EST Jul 19, 2017. license updated
    • intel academic license (license5) : unstable from 9:40am EST Jul 6, 2017 Due to license debug: stable from 11:00am EST Jul 6, 2017. Bug fixed
    • totalview and hyperworks academic license (license2) : unstable from 2:47pm EST Jun 26, 2017 Due to license migration: stable from Jun 28, 2017. Migration done
    • comsol academic license (license2) : unstable from 4:25pm EST May 16, 2017 Due to license maintenance: stable from 4:45pm EST May 16, 2017. New version installed
    • lstc academic license (license5) : unstable from 10:00am EST Jan 12, 2017 Due to license bug fixing: stable from 12:40pm EST Jan 12, 2017. Bug fixed
    • Ansys academic license (license2) : unstable from 10:15am EST Jan 11, 2017 Due to license maintenance: stable from 10:45am EST Jan 11, 2017. other maintenance expected
    • Altair Hyperworks academic license (license2) : stopped from 3:15am EST Dec 12, 2016 Due to license update: restarted by 3:25pm EST DEc 12, 2016: license updated
    • Intel compiler academic license (license5) : stopped from 11:20am EST Oct 27, 2016 Due to software update: restared by 11:40am EST Oct 27, 2016: software updated
    • abaqus academic license (license2) : stopped from 2:20pm EST Oct 26, 2016 Due to license renewal: restarted by 2:30pm EST Oct 26, 2016: license renewed
    • Matlab academic license (license5) : stopped from 4:10pm EST Sep 29, 2016 Due to license renewal: restored by 4:30pm EST Sep 29, 2016: license renewed.
    • starccm academic license (license2) : stopped from 2:00pm EST Sep 23, 2016 Due to license update: restored by 2:10pm EST Sep 23, 2016: license updated.
    • arcgis academic license (license3) : unstable from 3:45pm EST Aug 22, 2016 Due to troubleshooting: restored by 5:00pm EST Aug 22, 2016: Problem fixed, but test needed
    • Ansys academic license (license2) : unstable from 3:00am EST Aug 15, 2016 Due to troubleshooting: restored by 3:50pm EST Aug 15, 2016: Found another problem. Another attempt expected
    • Ansys academic license (license2) : stopped from 9:50am EST Aug 12, 2016 Due to software update: restarted on 11:10am EST Aug 12, 2016: Update failed. Another attempt expected
    • Comsol academic license (license2) : unstable from 3:20pm EST Aug 5, 2016 Due to troubleshooting: stable from 4:00pm EST Aug 5, 2016: Problem fixed
    • Comsol academic license (license2) : stopped from 11:10am EST Aug 5, 2016 Due to software update: restarted on 11:30am EST Aug 5, 2016. Update finished
    • arcGIS license: stopped from 11:20am EST Jul 21, 2016 Due to software update: restarted on 2:00pm EST Jul 21, 2016. Update finished
    • PGI statewide license: stopped from 11:20am EST Jul 20, 2016. Due to software update: restarted on 11:55am EST Jul 20, 2016. Update finished

     

     

    Software Forms

    Several of our software packages require that users complete a form that is then kept on record here before we can grant access to the software. OSC will provide you with the proper forms and instructions; please contact OSC Help noting which software package(s) you would like access.

    The following list includes third-party software packages that require additional forms be completed:

    ABAQUS

    Amber

    ANSYS

    CASINO

    CNS

    CSD

    FLUENT

    GATK

    Gaussian

    Hyperworks

    MATLAB DCS

    MuTect

    SIESTA

    STAR-CCM+ 

    Turbomole

    Requesting additional software

    If you would like to request that the center install (and purchase, if necessary) software, please complete the Request for Software Form. The return instructions are noted on the form.

    If you have licensed software and you need assistance installing the software and restricting it's use to your group, please contact OSC Help.

    Service: 

    Software List

    Ohio Supercomputer Center (OSC) has a variety of software applications to support all aspects of scientific research. We are actively updating this documentation to ensure it matches the state of the supercomputers. This page is currently missing some content; use module spider on each system for a comprehensive list of available software.

    Service: 

    ABAQUS

    ABAQUS is a finite element analysis program owned and supported by SIMULIA, the Dassault Systèmes brand for Realistic Simulation.

    Availability and Restrictions

    The available programs are ABAQUS/CAE, ABAQUS/Standard and ABAQUS/Explicit. The versions currently available at OSC are:

    Version Oakley Ruby Owens Notes
    6.11-2 X     Default version on Oakley prior to 09/15/2015
    6.12-pr3 X      
    6.12 X      
    6.13   X    
    6.14 X X* X  
    6.14-5 X      
    2016 X*   X* Version scheme has been changed. 
    2017 X      
    *: Current default version

    You can use  module spider abaqus  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    OSC's ABAQUS license can only be used for educational, institutional, instructional, and/or research purposes. Only users who are faculty, research staff or students at the following institutions are permitted to utilized OSC's license:

    • The Ohio State University
    • University of Toledo
    • University of Cincinnati
    • Miami University of Ohio
    • Wright State University
    • University of Akron

    Users from additional degree granting academic institutions may request to be added to this list per a cost by contacting OSC Help.

    The use of ABAQUS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction. 

     

    Access for Commerical Users

    Contact OSC Help for getting access to ABAQUS if you are a commercial user.

    Usage

    Token Usage

    ABAQUS software usage is monitored though a token-based license manager. This means very time you run a ABAQUS job, tokens are checked out from our pool for your tasks usage. To ensure your job is only started when its required ABAQUS tokens are available it is important to include a software flag within your job script's PBS directives.  A minimum of 5 tokens are required per a job, so a 1 node, 1 processor ABAQUS job would need the following PBS software flag: #PBS -l software=abaqus+5 . Jobs requiring more cores will need to request more tokens as calculated with the formula: M = int(5 x N^0.422) , where N is the total number of cores.  For common requests, you can refer to the following table:

             Cores            (nodes x ppn each):

    1 2 3 4 6 8 12 16 24 32 48
    Tokens needed: 5 6 7 8 10 12 14 16 19 21 25

    Usage on Oakley

    Set-up on Oakley

    To load the default version of ABAQUS module, use  module load abaqus . To select a particular software version, use   module load abaqus/version . For example, use  module load abaqus/6.12  to load ABAQUS version 6.12 on Oakley. 

    Using ABAQUS

    Example input data files are available with the ABAQUS release. The abaqus fetch utility is used to extract these input files for use. For example, to fetch input file for one of the sample problems including 4 input files, type:

    abaqus fetch job=knee_bolster

    abaqus fetch job=knee_bolster_ef1

    abaqus fetch job=knee_bolster_ef2

    abaqus fetch job=knee_bolster_ef3

    Also, use the abaqus help  utility is to list all the abaqus execution procedures.

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Oakley, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Assume that you have fetched the four input files above into your work directory (where you submit your job, represented by  $PBS_O_WORKDIR ). Below is the example batch script ( job.txt ) for a serial run:

    #PBS -N knee
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -l software=abaqus+5
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.inp $TMPDIR
    cd $TMPDIR
    #
    # Run ABAQUS
    #
    abaqus job=knee_bolster interactive
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt

    NOTE:

    • Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory ( $PBS_O_WORKDIR ) to  $TMPDIR , and copy your results back at the end of your script. Running your job on  $TMPDIR  ensures maximum efficiency.
    • The keyword interactive  is required in the execution line abaqus job=knee_bolster interactive for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword interactive in this case simply tells ABAQUS not to return until the simulation has completed.
    • The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. Although, it is better practice to designate the main input file explicitly by adding input=<my_input_file_name>.inp to the execution line so that it looks like abaqus job=knee_bolster input=<my_input_file_name>.inp interactive
    • Define nodes=1 (1<=ppn<=12 for Oakley) for a serial run.
    • If ppn>1, add cpus=<n> to the execution line, where n=ppn. This is how it should look like: abaqus job=test input=<my_input_file_name1>.inp cpus=<n> interactive
    Non-interactive Batch Job (Parallel Run)

    Below is the example batch script ( job.txt ) for a parallel run:

    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -N my_abaqus_job
    #PBS -l software=abaqus+19
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.inp $TMPDIR/
    cd $TMPDIR
    #
    # Run ABAQUS, note that in this case we have provided the names of the input files explicitly
    #
    abaqus job=test input=<my_input_file_name1>.inp cpus=24 interactive
    #
    # Now, move data back once the simulation has completed
    #
    mv * $PBS_O_WORKDIR
    

    NOTE:

    • Define nodes>1 for a parallel run. 1<=ppn<=12 for Oakley.
    • Specify  cpus=<n> in the execution line, where n=nodes*ppn.
    • Everything else is similar to the serial script above.

    Usage on Ruby

    Set-up on Ruby

    To load the default version of ABAQUS module, use  module load abaqus . To select a particular software version, use   module load abaqus/version . For example, use  module load abaqus/6.13  to load ABAQUS version 6.13 on Ruby. 

    Using ABAQUS

    Example input data files are available with the ABAQUS release. The abaqus fetch utility is used to extract these input files for use. For example, to fetch input file for one of the sample problems including 4 input files, type:

    abaqus fetch job=knee_bolster

    abaqus fetch job=knee_bolster_ef1

    abaqus fetch job=knee_bolster_ef2

    abaqus fetch job=knee_bolster_ef3

    Also, use the abaqus help  utility is to list all the abaqus execution procedures.

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Ruby, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00
    
    which gives you 20 cores ( -l nodes=1:ppn=20 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the wall time per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Assume that you have fetched the four input files above into your work directory (where you submit your job, represented by  $PBS_O_WORKDIR ). Below is the example batch script ( job.txt ) for a serial run:

    #PBS -N knee
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=20
    #PBS -l software=abaqus+17
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.inp $TMPDIR
    cd $TMPDIR
    #
    # Run ABAQUS
    #
    abaqus job=knee_bolster cpus=20 interactive
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt

    NOTE:

    • Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory ( $PBS_O_WORKDIR ) to  $TMPDIR , and copy your results back at the end of your script. Running your job on  $TMPDIR  ensures maximum efficiency.
    • The keyword interactive  is required in the execution line abaqus job=knee_bolster interactive for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword interactive in this case simply tells ABAQUS not to return until the simulation has completed.
    • The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. Although, it is better practice to designate the main input file explicitly by adding input=<my_input_file_name>.inp to the execution line so that it looks like abaqus job=knee_bolster input=<my_input_file_name>.inp interactive
    • Ruby has full node charging policy. Define nodes=1:ppn=20   for a serial run
    Non-interactive Batch Job (Parallel Run)

    Below is the example batch script ( job.txt ) for a parallel run:

    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=20
    #PBS -N my_abaqus_job
    #PBS -l software=abaqus+23
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.inp $TMPDIR/
    cd $TMPDIR
    #
    # Run ABAQUS, note that in this case we have provided the names of the input files explicitly
    #
    abaqus job=test input=<my_input_file_name1>.inp cpus=40 interactive
    #
    # Now, move data back once the simulation has completed
    #
    mv * $PBS_O_WORKDIR
    

    NOTE:

    • Define nodes>1 for a parallel run. 
    • Specify  cpus=<n> in the execution line, where n=nodes*ppn.
    • Everything else is similar to the serial script above.

    Usage on Owens

    Set-up on Owens

    To load the default version of ABAQUS module, use   module load abaqus  . To select a particular software version, use    module load abaqus/version  . For example, use   module load abaqus/2016   to load ABAQUS version 2016 on Owens. 

    Using ABAQUS

    Example input data files are available with the ABAQUS release. The  abaqus fetch  utility is used to extract these input files for use. For example, to fetch input file for one of the sample problems including 4 input files, type:

    abaqus fetch job=knee_bolster

    abaqus fetch job=knee_bolster_ef1

    abaqus fetch job=knee_bolster_ef2

    abaqus fetch job=knee_bolster_ef3

    Also, use the  abaqus help   utility is to list all the abaqus execution procedures.

    Batch Usage on Owens

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ABAQUS analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Owens, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores (  -l nodes=1:ppn=28  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Assume that you have fetched the four input files above into your work directory (where you submit your job, represented by   $PBS_O_WORKDIR  ). Below is the example batch script (  job.txt  ) for a serial run:

    #PBS -N knee
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -l software=abaqus+5
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.inp $TMPDIR
    cd $TMPDIR
    #
    # Run ABAQUS
    #
    abaqus job=knee_bolster interactive
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the  job.txt   file with the command:  qsub job.txt

    NOTE:

    • Make sure to copy all the files needed (input files, restart files, user subroutines, python scripts etc.) from your work directory (  $PBS_O_WORKDIR  ) to   $TMPDIR  , and copy your results back at the end of your script. Running your job on   $TMPDIR   ensures maximum efficiency.
    • The keyword  interactive   is required in the execution line  abaqus job=knee_bolster interactive  for the following reason: If left off, ABAQUS will background the simulation process. Backgrounding a process in the OSC environment will place it outside of the batch job and it will receive the default 1 hour of CPU time and corresponding default memory limits. The keyword  interactive  in this case simply tells ABAQUS not to return until the simulation has completed.
    • The name of the input file is sometimes omitted in the execution line, which may work fine if you've copied only the input files for one specific model. Although, it is better practice to designate the main input file explicitly by adding  input=<my_input_file_name>.inp  to the execution line so that it looks like abaqus job=knee_bolster input=<my_input_file_name>.inp interactive
    • Define  nodes=1  (1<=ppn<=28 for Owens) for a serial run.
    • If ppn>1, add  cpus=<n>  to the execution line, where n=ppn. This is how it should look like: abaqus job=test input=<my_input_file_name1>.inp cpus=<n> interactive
    Non-interactive Batch Job (Parallel Run)
    Note: abaqus will not run correctly in parallel with input files in $TMPDIR!  Use the scratch file system.

    Below is the example batch script (  job.txt  ) for a parallel run:

    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=28:pfsdir
    #PBS -N my_abaqus_job
    #PBS -l software=abaqus+27
    #PBS -j oe
    #
    # The following lines set up the ABAQUS environment
    #
    module load abaqus
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    #
    # Cope input files to /fs/scratch and run Abaqus there
    #
    cp *.inp $PFSDIR
    cd $PFSDIR
    #
    # Run ABAQUS, note that in this case we have provided the names of the input files explicitly
    #
    abaqus job=test input=<my_input_file_name1>.inp cpus=56 interactive
    #
    # Now, move data back once the simulation has completed
    #
    mv * $PBS_O_WORKDIR
    

     

    NOTE:

    • Define nodes>1 for a parallel run. 1<=ppn<=28 for Owens.
    • Specify   cpus=<n>  in the execution line, where n=nodes*ppn.
    • Everything else is similar to the serial script above.

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    ABINIT

    ABINIT is a package whose main program allows one to find the total energy, charge density and electronic structure of systems made of electrons and nuclei (molecules and periodic solids) within Density Functional Theory (DFT), using pseudopotentials and a planewave or wavelet basis. ABINIT also includes options to optimize the geometry according to the DFT forces and stresses, or to perform molecular dynamics simulations using these forces, or to generate dynamical matrices, Born effective charges, and dielectric tensors, based on Density-Functional Perturbation Theory, and many more properties.

    Availability and Restrictions

    The following versions of ABINIT are available on OSC clusters:

    Version Oakley
    7.10.5 X*
    *: Current default version

    You can use  module spider abinit  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    ABINIT is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of ABINIT on Oakley, run the following command:   module load abinit. The default version will be loaded. To select a particular ABINIT version, use   module load abinit/version  . For example, use  module load abinit/7.10.5 to load ABINIT 7.10.5 on Oakley.

    Further Reading

    Service: 

    AMBER

    The Assisted Model Building with Energy Refinement (AMBER) package contains many molecular simulation programs targeted at biomolecular systems. A wide variety of modelling techniques are available. It generally scales well on modest numbers of processors, and the GPU enabled CUDA programs are very efficient.

    Availability and Restrictions

    AMBER is available on Oakley, Ruby, and Owens Clusters. The following versions are currently available at OSC (S means serial executables, P means parallel, C means CUDA, i.e., GPU enabled, and M means MIC, i.e., Xeon Phi enabled):

    Version Oakley ruby Owens notes
    11 SPC     Default version on Oakley prior to 09/15/2015
    12 SPC      
    14 SPC SPC   Default version on Oakley and Ruby prior to 02/21/2017
    16 SPC* SPCM* SPC*  
    *: Current default version

    You can use module spider amber to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    OSC's Amber is available to not-for-profit OSC users; simply contact OSC Help to request the appropriate form for access.

    Access for Commerical Users

    For-profit OSC users must obtain their own Amber license. 

    Usage on Oakley

    Set-up

    To load the default version of AMBER module, use  module load amber  . To select a particular software version, use   module load amber/version . For example, use  module load amber/14   to load AMBER version 14. 

    Using AMBER

    A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

    tleap
    

    Parallel Amber programs must be run in a batch environment with mpiexec, e.g.:

    mpiexec pmemd.MPI
    

    The CUDA Amber programs, which must be run in a batch environment, are recommended due to their computational performance; e.g.:

    pmemd.cuda
    

    For supported features of the GPU accelerated version of PMEMD see Amber GPU Support.

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

    ~srb/workshops/compchem/amber/
    

    Below is the example batch script (  job.txt  ) for a serial run:

    # AMBER Example Batch Script for the Basic Tutorial in the Amber manual
    #PBS -N 6pti
    #PBS -l nodes=1:ppn=1
    #PBS -l walltime=0:20:00
    module load amber
    # Use TMPDIR for best performance.
    cd $TMPDIR
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cp -p $PBS_O_WORKDIR/6pti.prmtop .
    cp -p $PBS_O_WORKDIR/6pti.prmcrd .
    # Running minimization for BPTI
    cat << eof > min.in
    # 200 steps of minimization, generalized Born solvent model
    &cntrl
    maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
    /
    eof
    sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
    cp -p min.in 6pti.min1.out 6pti.min1.xyz $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

    Usage on Ruby

    Set-up

    To load the default version of AMBER module, use  module load amber  . To select a particular software version, use   module load amber/version . For example, use  module load amber/14   to load AMBER version 14. 

    Using AMBER

    A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

    tleap
    

    Parallel Amber programs must be run in a batch environment with mpiexec, e.g.:

    mpiexec pmemd.MPI
    

    The CUDA Amber programs, which must be run in a batch environment, are recommended due to their computational performance; e.g.:

    pmemd.cuda
    

    For supported features of the GPU accelerated version of PMEMD see Amber GPU Support.

    The MIC, i.e., Xeon Phi enabled, Amber programs must be run in a batch environment.  The Xeon Phi coprocessors on Ruby were removed from service. Please contact OSC Help if you have any questions or want to get access to such resources. 

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00
    
    which gives you 20 cores ( -l nodes=1:ppn=20 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

    ~srb/workshops/compchem/amber/
    

    Below is the example batch script (  job.txt  ) for a serial run:

    # AMBER Example Batch Script for the Basic Tutorial in the Amber manual
    #PBS -N 6pti
    #PBS -l nodes=1:ppn=20
    #PBS -l walltime=0:20:00
    module load amber
    # Use TMPDIR for best performance.
    cd $TMPDIR
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cp -p $PBS_O_WORKDIR/6pti.prmtop .
    cp -p $PBS_O_WORKDIR/6pti.prmcrd .
    # Running minimization for BPTI
    cat << eof > min.in
    # 200 steps of minimization, generalized Born solvent model
    &cntrl
    maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
    /
    eof
    sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
    cp -p min.in 6pti.min1.out 6pti.min1.xyz $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

    Usage on Owens

    Set-up

    To load the default version of AMBER module, use  module load amber  . To select a particular software version, use   module load amber/version . For example, use  module load amber/16   to load AMBER version 16. 

    Using AMBER

    A serial Amber program in a short duration run can be executed interactively on the command line, e.g.:

    tleap
    

    Parallel Amber programs must be run in a batch environment with mpiexec, e.g.:

    mpiexec pmemd.MPI
    

     

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your AMBER simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores ( -l nodes=1:ppn=28 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Amber input files are available here:

    ~srb/workshops/compchem/amber/
    

    Below is the example batch script (  job.txt) for a serial run:

    # AMBER Example Batch Script for the Basic Tutorial in the Amber manual
    #PBS -N 6pti
    #PBS -l nodes=1:ppn=28
    #PBS -l walltime=0:20:00
    module load amber
    # Use TMPDIR for best performance.
    cd $TMPDIR
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cp -p $PBS_O_WORKDIR/6pti.prmtop .
    cp -p $PBS_O_WORKDIR/6pti.prmcrd .
    # Running minimization for BPTI
    cat << eof > min.in
    # 200 steps of minimization, generalized Born solvent model
    &cntrl
    maxcyc=200, imin=1, cut=12.0, igb=1, ntb=0, ntpr=10,
    /
    eof
    sander -i min.in -o 6pti.min1.out -p 6pti.prmtop -c 6pti.prmcrd -r 6pti.min1.xyz
    cp -p min.in 6pti.min1.out 6pti.min1.xyz $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

    Further Reading

    Service: 

    ANSYS

    ANSYS offers a comprehensive software suite that spans the entire range of physics, providing access to virtually any field of engineering simulation that a design process requires. Supports are provided by ANSYS, Inc

    Availability and Restrictions

    OSC has academic license of structural-fluid dynamics academic products, which offer structural mechanics, explicit dynamics, fluid dynamics and thermal simulation capabilities. These bundles also include ANSYS Workbench, relevant CAD import tools, solid modeling and meshing, and High Performance Computing (HPC) capability. See "Academic Research -> ANSYS Academic Research Mechanical and CFD" in this table for all available products at OSC.

    Access for Academic Users

    OSC has an "Academic Research " license for ANSYS. This allows for academic use of the software by Ohio faculty and students, with some restrictions. To view current ANSYS node restrictions, please see ANSYS's Terms of Use.

    Use of ANSYS products at OSC for academic purposes requires validation. Please contact OSC Help for further instruction.

    Access for Commerical Users

    Contact OSC Help for getting access to ANSYS if you are a commerical user.

    Usage

    For more information on how to use each ANSYS product at OSC systems, refer to its documentation page provided at the end of this page.

    Note

    Due to the way our Fluent and ANSYS modules are configured, simultaneously loading multiple of either module will cause a cryptic error. The most common case of this happening is when multiple of a users jobs are started at the same time and all load the module at once. In order for this error to manifest, the modules have to be loaded at precisely the same time; a rare occurrence, but a probable occurrence over the long term.

    If you encounter this error you are not at fault. Please resubmit the failed job(s).

    If you frequently submit large amounts of Fluent or ANSYS jobs, we recommend you stagger your job submit times to lower the chances of two jobs starting at the same time, and hence loading the module at the same time. Another solution is to establish job dependencies between jobs, so jobs will only start one after another. To do this, you would add the PBS directive:

    #PBS -W after:jobid

    To jobs you want to only start after another job has started. You would replace jobid with the job ID of the job to wait for. If you have additional questions, please contact OSC Help.

    Further Reading

    See Also

    Service: 

    ANSYS Mechanical

    ANSYS Mechanical is a finite element analysis (FEA) tool that enables you to analyze complex product architectures and solve difficult mechanical problems. You can use ANSYS Mechanical to simulate real world behavior of components and sub-systems, and customize it to test design variations quickly and accurately.

    Availability and Restrictions

    ANSYS Mechanical is available on the Oakley Cluster. The versions currently available at OSC are:

    Version Oakley Notes
    14.0 X  
    14.5.7 X Default version on Oakley prior to 09/15/2015
    16.0 X*  
    17.0 X  
    17.1 X  
    17.2 X  
    18.0 X  
    18.1 X  
    *: Current default version

    You can use module spider ansys to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    Use of ANSYS for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

    Access for Commerical Users

    Contact OSC Help for getting access to ANSYS if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version of ANSYS module, use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/13  to load ANSYS version 13.0 on Oakley. 

    Using ANSYS Mechanical

    Following a successful loading of the ANSYS module, you can access the ANSYS Mechanical commands and utility programs located in your execution path:

    ansys <switch options> <file>
    

    The ANSYS Mechanical command takes a number of Unix-style switches and parameters.

    The -j Switch

    The command accepts a -j switch. It specifies the "job id," which determines the naming of output files. The default is the name of the input file.

    The -d Switch

    The command accepts a -d switch. It specifies the device type. The value can be X11, x11, X11C, x11c, or 3D.

    The -m Switch

    The command accepts a -m switch. It specifies the amount of working storage obtained from the system. The units are megawords.

    The memory requirement for the entire execution will be approximately 5300000 words more than the -m specification. This is calculated for you if you use ansnqs to construct an NQS request.

    The -b [nolist] Switch

    The command accepts a -b switch. It specifies that no user input is expected (batch execution).

    The -s [noread] Switch

    The command accepts a -s switch. By default, the start-up file is read during an interactive session and not read during batch execution. These defaults may be changed with the -s command line argument. The noread option of the -s argument specifies that the start-up file is not to be read, even during an interactive session. Conversely, the -s argument with the -b batch argument forces the reading of the start-up file during batch execution.

    The -g [off] Switch

    The command accepts a -g switch. It specifies that the ANSYS graphical user interface started automatically.

    ANSYS Mechanical parameters

    ANSYS Mechanical parameters may be assigned values on the command. The parameter must be at least two characters long and must be a legal parameter name. The ANSYS Mechanical parameter that is to be assigned a value should be given on the command line with a preceding dash (-), a space immediately after, and the value immediately after the space:

    module load ansys
    ansys -pval1 -10.2 -EEE .1e6
    sets pval1 to -10.2 and EEE to 100000
    

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your ANSYS Mechanical analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    Interactive mode is similar to running ANSYS Mechanical on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run ANSYS Mechanical interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

    To run interactive ANSYS Mechanical, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. For example, the following command requests one whole node with 12 cores ( -l nodes=1:ppn=12 ), for a walltime of 1 hour (  -l walltime=1:00:00 ), with one ANSYS license:

    qsub -I -X -l nodes=1:ppn=12 -l walltime=1:00:00 -l software=ansys+1
    

    You may adjust the numbers per your need. This job will queue until resources becomes available. Once the job is started, you're automatically logged in on the compute node; and you can launch ANSYS Mechanical and start the graphic interface with the following commands:

    module load ansys
    ansys -g
    
    Non-interactive Batch Job (Serial Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. For a given model, prepare the input file with ANSYS Mechanical commands (named  ansys.in  for example) for the batch run. Below is the example batch script (   job.txt ) for a serial run:

    #PBS -N ansys_test  
    #PBS -l walltime=30:00:00  
    #PBS -l nodes=1:ppn=1
    #PBS -l software=ansys+1  
    #PBS -j oe
    cd $TMPDIR  
    cp $PBS_O_WORKDIR/ansys.in .    
    module load ansys  
    ansys < ansys.in   
    cp <output files> $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the  job.txt  file with the command:  qsub job.txt .

    Non-interactive Batch Job (Parallel Run)

    To take advantage of the powerful compute resources at OSC, you may choose to run distributed ANSYS Mechanical for large problems. Multiple nodes and cores can be requested to accelerate the solution time. Note that you'll need to change your batch script slightly for distributed runs.

    Starting from September 15, 2015, a job using HPC tokens (with "ansyspar" flag) should be submitted to Oakley clusters due to scheduler issue.

    For distributed ANSYS Mechanical jobs using one node (nodes=1), the number of processors needs to be specified in the command line with options '-dis -np':

    #PBS -N ansys_test 
    #PBS -l walltime=3:00:00 
    #PBS -l nodes=1:ppn=12
    #PBS -l software=ansys+1%ansyspar+8
    ...
    ansys -dis -np 12 < ansys.in  
    ...
    

    Notice that in the script above, the ansys parallel license is requested as well as ansys license in the format of

    #PBS -l software=ansys+1%ansyspar+n
    

    where n=m-4, with m being the total cpus called for this job. This line is necessary when the total cpus called is greater than 4 (m>4), which applies for the parallel example below as well.

    For distributed jobs requesting multiple nodes, you need to specify the number of processors for each node in the command line. This information can be obtained from $PBS_NODEFILE. The following shows changes in the batch script if 2 nodes on Oakley are requested for a parallel ANSYS Mechanical job:

    #PBS -N ansys_test 
    #PBS -l walltime=3:00:00 
    #PBS -l nodes=2:ppn=12
    #PBS -l software=ansys+1%ansyspar+20
    ...
    export MPI_WORKDIR=$PWD
    machines=`uniq -c ${PBS_NODEFILE} | awk '{print $2 ":" $1}' | paste -s -d ':'`
    ansys -dis -machines $machines < ansys.in  
    ...
    pbsdcp -g '<output files>' $PBS_O_WORKDIR
    

    The 'pbsdcp -g' command in the last line in the script above makes sure that all result files generated by different compute nodes are copied back to the work directory.

    Further Reading

    See Also

    Service: 

    CFX

    ANSYS CFX (called CFX hereafter) is a computational fluid dynamics (CFD) program for modeling fluid flow and heat transfer in a variety of applications.

    Availability and Restrictions

    CFX is available on the Oakley Cluster. The versions currently available at OSC are:

    VERSION OAKLEY notes
    14.5.7 X  
    15.0.7 X  
    16.0 X*  
    17.0 X  
    17.1 X  
    17.2 X  
    18.0 X  
    18.1 X  
    *: Current default version

    You can use module spider fluent  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

    Currently, there are in total 50 ANSYS CFD base license tokens and 256 HPC tokens for academic users. These base tokens are shared by available ANSYS CFD related projusts (see "Academic Research -> ANSYS Academic Research CFD" in this table for details). These HPC tokens are shared with all ANSYS products we have at OSC. A base license token will allow CFX to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial CFX job with 1 core will need 1 base license token while a parallel CFX job with 12 cores will need 1 base license token and 8 HPC tokens.

    Access for Commerical Users

    Contact OSC Help for getting access to CFX if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version, use  module load fluent . To select a particular software version, use   module load fluent/version . For example, use  module load fluent/16.0  to load CFX version 16.0 on Oakley. 

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    Interactive mode is similar to running CFX on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run CFX interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

    To run interactive CFX GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please follwoing the steps below to use CFX GUI interactivly:

    1. Ensure that your SSH client software has X11 forwarding enabled
    2. Connect to Oakley system
    3. Request an interactive job. The command below will request one whole node with 12 cores (  -l nodes=1:ppn=12 ), for a walltime of one hour ( -l walltime=1:00:00 ), with one ANSYS CFD license (modify as per your own needs):
      qsub -I -X -l nodes=1:ppn=12 -l walltime=1:00:00 -l software=ansys+1
      
    4. Once the interactive job has started, run the following commands to setup and start the CFX GUI:

      module load fluent
      cfx5 
      
    Non-interactive Batch Job (Serial Run Using 1 Base Token)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Below is the example batch script (  job.txt ) for a serial run with an input file test.def ) on Glenn:

    #PBS -N serialjob_cfx
    #PBS -l walltime=1:00:00
    #PBS -l software=ansys+1
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    #PBS -S /bin/bash
    #Set up CFX environment.
    module load fluent
    #'cd' directly to your working directory
    cd $PBS_O_WORKDIR
    #Copy CFX files like .def to $TMPDIR and move there to execute the program
    cp test.def $TMPDIR/
    cd $TMPDIR
    #Run CFX in serial with test.def as input file
    cfx5solve -batch -def test.def 
    #Finally, copy files back to your home directory
    cp  * $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

    Non-interactive Batch Job (Parallel Execution using HPC token)

    CFX can be run in parallel, but it is very important that you read the documentation in the CFX Manual on the details of how this works.

    In addition to requesting the base license token ( -l software=fluent+1 ), you need to request copies of the ansyspar license, i.e., HPC tokens (-l software=ansys+1%ansyspar+[n] ), where [n] is equal to the number of cores you requested minus 4.

    Parallel jobs have to be submitted on Oakley via the batch system. An example of the batch script follows:

    #PBS -N paralleljob_cfx
    #PBS -l walltime=10:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -l software=ansys+1%ansyspar+20
    #PBS -j oe
    #PBS -S /bin/bash
    #Set up CFX environment.
    module load fluent
    #'cd' directly to your working directory
    cd $PBS_O_WORKDIR
    #Copy CFX files like .def to $TMPDIR and move there to execute the program
    cp test.def $TMPDIR/
    cd $TMPDIR
    #Convert PBS_NODEFILE information into format for CFX host list
    nodes=`cat $PBS_NODEFILE`
    nodes=`echo $nodes | sed -e 's/ /,/g'`
    #Run CFX in parallel with new.def as input file
    #if multiple nodes
    cfx5solve -batch -def test.def  -par-dist $nodes -start-method "Platform MPI Distributed Parallel"
    #if one node
    #cfx5solve -batch -def test.def -par-dist $nodes -start-method "Platform MPI Local Parallel"
    #Finally, copy files back to your home directory
    cp  * $PBS_O_WORKDIR
    

    Further Reading

    Service: 

    FLUENT

    ANSYS FLUENT (called FLUENT hereafter) is a state-of-the-art computer program for modeling fluid flow and heat transfer in complex geometries.

    Availability and Restrictions

    FLUENT is available on the Oakley Cluster. The versions currently available at OSC are:

    Version Oakley NOTEs
    14 X  
    14.5.7 X  
    15.0.7 X  
    16.0 X*  
    17.0 X  
    17.1 X  
    17.2 X  
    18.0 X  
    18.1 X  
    *: Current default version

    You can use module spider fluent  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

    Currently, there are in total 50 ANSYS CFD base license tokens and 256 HPC tokens for academic users. These base tokens are shared by available ANSYS CFD related projusts (see "Academic Research -> ANSYS Academic Research CFD" in this table for details). These HPC tokens are shared with all ANSYS products we have at OSC. A base license token will allow FLUENT to use up to 4 cores without any additional tokens. If you want to use more than 4 cores, you will need an additional "HPC" token per core. For instance, a serial FLUENT job with 1 core will need 1 base license token while a parallel FLUENT job with 12 cores will need 1 base license token and 8 HPC tokens.

    Access for Commerical Users

    Contact OSC Help for getting access to FLUENT if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version of FLUENT module, use  module load fluent . To select a particular software version, use   module load fluent/version . For example, use  module load fluent/16.0  to load FLUENT version 16.0 on Oakley. 

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your FLUENT analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    Interactive mode is similar to running FLUENT on a desktop machine in that the graphical user interface will be sent from OSC and displayed on the local machine. Interactive jobs are run on compute nodes of the cluster, by turning on X11 forwarding. The intention is that users can run FLUENT interactively for the purpose of building their model and preparing their input file. Once developed this input file can then be run in no-interactive batch mode.

    To run interactive FLUENT GUI, a batch job need to be submitted from the login node, to request necessary compute resources, with X11 forwarding. Please follwoing the steps below to use FLUENT GUI interactivly:

    1. Ensure that your SSH client software has X11 forwarding enabled
    2. Connect to Oakley system
    3. Request an interactive job. The command below will request one whole node with 12 cores (  -l nodes=1:ppn=12 ), for a walltime of one hour ( -l walltime=1:00:00 ), with one FLUENT license (modify as per your own needs):
      qsub -I -X -l nodes=1:ppn=12 -l walltime=1:00:00 -l software=ansys+1
      
    4. Once the interactive job has started, run the following commands to setup and start the FLUENT GUI:

      module load fluent
      fluent 
      
    Non-interactive Batch Job (Serial Run Using 1 Base Token)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Below is the example batch script (  job.txt ) for a serial run with an input file run.input ) on Oakley:

    #PBS -N serial_fluent
    #PBS -l walltime=5:00:00 
    #PBS -l nodes=1:ppn=1
    #PBS -l software=ansys+1
    #PBS -j oe
    #
    # The following lines set up the FLUENT environment
    #
    module load fluent
    #
    # Move to the directory where the job was submitted from
    # You could also 'cd' directly to your working directory
    cd $PBS_O_WORKDIR
    #
    # Copy files to $TMPDIR and move there to execute the program
    #
    cp test_input_file.cas test_input_file.dat run.input $TMPDIR
    cd $TMPDIR
    #
    # Run fluent
    fluent 3d -g < run.input  
    #
    # Where the file 'run.input' contains the commands you would normally
    # type in at the Fluent command prompt.
    # Finally, copy files back to your home directory
    cp *   $PBS_O_WORKDIR  
    

    As an example, your run.input file might contain:

    file/read-case-date test_input_file.cas 
    solve/iterate 100
    file/write-case-data test_result.cas
    file/confirm-overwrite yes    
    exit  
    yes  
    

    In order to run it via the batch system, submit the job.txt  file with the command: qsub job.txt  

    Non-interactive Batch Job (Parallel Execution using HPC token)

    FLUENT can be run in parallel, but it is very important that you read the documentation in the FLUENT Manual on the details of how this works.

    In addition to requesting the FLUENT base license token ( -l software=ansys+1 ), you need to request copies of the ansyspar license, i.e., HPC tokens (-l software=ansys+1%ansyspar+[n] ), where [n] is equal to the number of cores you requested minus 4.

    Parallel jobs have to be submitted on Oakley via the batch system. An example of the batch script follows:

    #PBS -N parallel_fluent   
    #PBS -l walltime=1:00:00   
    #PBS -l nodes=2:ppn=12
    #PBS -j oe
    #PBS -l software=ansys+1%ansyspar+20
    #PBS -S /bin/bash
    set echo on   
    hostname   
    #   
    # The following lines set up the FLUENT environment   
    #   
    module load fluent
    #   
    # Move to the directory where the job was submitted from and   
    # create the config file for socket communication library   
    #   
    cd $PBS_O_WORKDIR   
    #   
    # Create list of nodes to launch job on   
    rm -f pnodes   
    cat  $PBS_NODEFILE | sort > pnodes   
    export ncpus=`cat pnodes | wc -l`   
    #   
    #   Run fluent   
    fluent 3d -t$ncpus -pinfiniband.ofed -cnf=pnodes -g < run.input 
    

    Further Reading

    See Also

    Service: 

    Workbench Platform

    ANSYS Workbench platform is the backbone for delivering a comprehensive and integrated simulation system to users. See ANSYS Workbench platform for more information. 

    Availability and Restrictions

    ANSYS Workbench is available on Oakley Cluster. The versions currently available at OSC are:

    Version Oakley Notes
    14.5.7 SF X  
    CFD X
    15.0.7 SF X  
    CFD X
    16.0 SF X*  
    CFD X*
    17.0 SF X  
      CFD X  
    17.1 SF X  
      CFD X  
    17.2 SF X  
      CFD X  
    18.0 SF X  
      CFD X  
    18.1 SF X  
      CFD X  
    *: Current default version

    Note:

    • SF: Structural-Fluid dynamics related applications. See "Academic Research -> ANSYS Academic Research Mechanical and CFD" in this table for all available products
    • CFD: CFD related applications. See "Academic Research -> ANSYS Academic Research CFD" in this table for all available products

    You can use module spider ansys  to view available modules for a given machine if you want to use structural-fluid dynamics related applications or module spider fluent  to view available modules for a given machine if you want to use CFD related applications. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    Use of ANSYS products for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

    Access for Commerical Users

    Contact OSC Help for getting access to ANSYS if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up for Structural-Fluid dynamics related applications

    To load the default version , use  module load ansys . To select a particular software version, use   module load ansys/version . For example, use  module load ansys/15.0.7   to load version 15.0.7 on Oakley. After the module is loaded, use the following command to open Workbench GUI:

    runwb2
    

    Set-up for CFD related applications

    To load the default version , use  module load fluent  . To select a particular software version, use   module load ansys/fluent  . For example, use  module load fluent/15.0.7   to load version 15.0.7 on Oakley. After the module is loaded, use the following command to open Workbench GUI:

    runwb2
    

    Further Reading

    See Also

    Service: 

    Allinea

    Allinea software analyzes how HPC software runs. It consists of two applications, Allinea Performance Reports and Allinea MAP: 

    • Allinea Performance Reports: simple tool to generate a single-page HTML or plain text report that presents overall performance characteristics of HPC applications.
    • Allinea MAP: HPC application profiler with easy-to-use GUI environment.

    Availability & Restrictions

    Versions

    The following versions of Allinea are available on OSC clusters:

    version oakley Ruby Owens
    5.1 X X  
    6.0.1 X X  
    6.0.6     X
    6.1.1   X        
    7.0  X*  X*

     X*

    * Current default version

    Access

    Allinea Performance Reports and Allinea MAP are available to all OSC users.

    Usage

    Allinea Performance Reports

    Allinea Performance Reports analyzes and documents information on CPU, MPI, I/O, and Memory performance characteristics of HPC software, even third party code, to aid understanding about the overall performance. Although it should not be used all the time, Allinea Performance Reports is recommended to OSC users as a viable option to analyze how an HPC application runs. View an example report to navigate the format of a typical report. For more example reports, visit Characterization of HPC Codes and Problems on Allinea's website.

    For usage instructions and more information, read Allinea Performance Reports.

    Allinea MAP

    Allinea MAP produces a more detailed profile of HPC software. Unlike Allinea Performance Reports, you must have the source code to run Allinea MAP because its analysis details the software line-by-line. For more features and benefits, visit Profiler Features and Benefits on Allinea's website. 

    For usage instructions and more information, read Allinea MAP.

    Troubleshooting

    Using Allinea software with MVAPICH2

    This note from Allinea's Getting Started Guide applies to both perf-report and MAP:

    Some MPIs, most notably MVAPICH, are not yet supported by Allinea's Express Launch mode
    (in which you can just put “perf-report” in front of an existing mpirun/mpiexec line). These can
    still be measured using the Compatibility Launch mode.

    Instead of this Express Launch command:

    perf-report mpiexec <mpi args> <program> <program args> # BAD
    

    Use the compatibility launch version instead:

    perf-report -n <num procs> --mpiargs="<mpi args>" <program> <program args>
    

    Further Reading

    See Also

    Documentation Attachment: 
    Service: 
    Fields of Science: 

    Allinea Performance Reports

    Allinea Performance Reports is a simple tool used to generate a single-page HTML or plain text report that presents the overall performance characteristics of HPC applications. It supports pthreads, OpenMP, or MPI code on CPU, GPU, and MIC based architectures.

    Availability and Restrictions

    Versions

    The versions currently available at OSC are

    VERSION OAKLEY RUBY OWENS
    5.1 X X  
    6.0.1 X X  
    6.0.6     X      
    6.1.1   X  
    7.0  X*  X*  X*
     
    * Current default version

    You can use module spider allinea to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Allinea Performance Reports is available to all OSC users. We have 64 seats with 64 HPC tokens. Users can monitor the license status here.

    Usage

    Set-up

    To load the module for the Allinea Performance Reports default version, use module load allinea. To select a particular software version, use module load allinea/version. For example, use module load allinea/6.0 to load Allinea Performance Reports version 6.0, provided the version is available on the OSC cluster in use.

    Using Allinea Performance Reports

    You can use your regular executables to generate performance reports. The program can be used to analyze third-party code as well as code you develop yourself. Performance reports are normally generated in a batch job.

    To generate a performance report for an MPI program:

    module load allinea
    perf-report -np <num procs> --mpiargs="<mpi args>" <program> <program args>
    

    where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

    For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

    perf-report -np 12 wave_c
    

    To generate a performance report for a non-MPI program:

    module load allinea
    perf-report --no-mpi <program> <program args>
    

    The performance report is created in both html and plain text formats. The file names are based on the executable name, number of processes, date and time, for example,  wave_c_12p_2016-02-05_12-46.html. To open the report in html format use

    firefox wave_c_12p_2016-02-05_12-46.html
    

    For more details, download the Allinea Performance Reports User Guide.

    Performance Reports with GPU

    Allinea Performance Reports can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Allinea Performance Reports with

    perf-report {executable}
    

    For more information, please read the section 6.9 of the Allinea Performance Reports User Guide.

    Further Reading

    See Also

    Documentation Attachment: 
    Service: 

    Allinea MAP

    Allinea MAP is a full scale profiler for HPC programs. We recommend using Allinea MAP after reviewing reports from Allinea Performance Reports. MAP supports pthreads, OpenMP, and MPI software on CPU, GPU, and MIC based architectures.

    Availability & Restrictions

    Versions

    The Allinea MAP versions currently available at OSC are

    VERSION OAKLEY RUBY OWENS
    5.1 X X  
    6.0.1 X X  
    6.0.6     X      
    6.1.1   X  
    7.0  X*  X*

     X*

    * Current default version

    You can use module spider allinea to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Allinea MAP is available to all OSC users. We have 64 seats with 80 HPC tokens. Users can monitor the Allinea License Server Status.

    Usage

    Set-up

    To load the default version of the Allinea MAP module, use module load allinea. To select a particular software version, use module load allinea/version. For example, use module load allinea/6.0 to load allinea MAP version 6.0, provided the version is available on the cluster in use. 

    Using Allinea MAP

    Profiling HPC software with Allinea MAP typically involves three steps: 

    1. Prepare the executable for profiling.

    Regular executables can be profiled with Allinea MAP, but source code line detail will not be available. You need executables with debugging information to view source code line detail: re-compile your code with a -g  option added among the other appropriate compiler options. For example:

    mpicc wave.c -o wave -g -O3
    

    This executable built with the debug flag can be used for Allinea Performance Reports as well.

    Note: The -g  flag turns off all optimizations by default. For profiling your code you should use the same optimizations as your regular executable, so explicitly include the -On flag, where n is your normal level of optimization, typically -O2 or -O3, as well as any other compiler optimization options.

    2. Run your code to produce the profile data file (.map file).

    Profiles are normally generated in a batch job.  To generate a MAP profile for an MPI program:

    module load allinea
    map --profile -np <num proc> --mpiargs="<mpi args>" <program> <program args>

    where <num procs> is the number of MPI processes to use, <mpi args> represents arguments to be passed to mpiexec (other than -n or -np), <program> is the executable to be run and <program args> represents arguments passed to your program.

    For example, if you normally run your program with mpiexec -n 12 wave_c, you would use

    map --profile -np 12 wave_c

    To profile a non-MPI program:

    module load allinea
    map --profile --no-mpi <program> <program args>

    The profile data is saved in a .map file in your current directory.

    As a result of this step, a .map file that is the profile data file is created in your current directory. The file name is based on the executable name, number of processes, date and time, for example,  wave_c_12p_2016-02-05_12-46.map.

    For more details on using Allinea MAP, refer to the Allinea Forge User Guide.

    3. Analyze the profile data file using either the Allinea local client or the MAP GUI.

    You can open the profile data file using a client running on your local desktop computer. For client installation and usage instructions, please refer to the section: Client Download and Setup. This option typically offers the best performance.

    Alternatively, you can run MAP in interactive mode, which launches the graphical user interface (GUI).  For example:

    map wave_c_12p_2016-02-05_12-46.map

    For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

    MAP with GPU

    Allinea MAP can be used for CUDA codes. If you have an executable compiled with the CUDA library, you can launch Allinea MAP with

    map {executable}
    

    For more information, please read the chapter 15 of the Allinea Forge User Guide.

    Client Download and Setup

    1. Download the client.

    To download the client, go to the Allinea website and choose the appropriate Allinea Forge remote client download for Windows, Mac, or Linux. For Windows and Mac, just double click on the downloaded file and allow the installer to run. For Linux, extract the tar file using the command tar -xf file_name and run the installer in the extracted file directory with ./installer. Please contact OSC Help, if you have any issues on downloading the client.

    Note:  Allinea was recently bought out by Arm.  The Allinea Forge client can be downloaded from https://developer.arm.com/products/software-development-tools/hpc/downloads/download-arm-forge/older-versions-of-remote-client-for-arm-forge.
    2. Configure the client.

    After installation, you can configure the client as follows:

    • Open the client program. For Windows or Mac, just click the desktop icon or navigate to the application through its file path. For Linux use the command {allinea-forge-path}/bin/map.

    • Once the program is launched, select Allinea MAP in the left column.
    • In the Remote Launch drop down menu, select "Configure...".
    • Click Add to create a new profile for your login.
    • In the Host Name section, type your ssh connection. For example: "username@ruby.osc.edu".
    • For Remote Installation Directory, type /usr/local/allinea/forge-{version}, specifying the Allinea Forge version number that created the data profile file you are attempting to view. For example, /usr/local/allinea/forge-7.0 for Allinea Forge version 7.0.
    • You can test your login information by clicking Test Remote Launch. It will ask your password. Use the same password for the cluster login.
    • Close the Configure window. You will see a new option under the Remote Launch drop down menu for the host name you entered. Select your profile and login with your password. 
    • If the login was successful, then you should see License Serial:XXX in the bottom left corner of the window.

    This login configuration is needed only for the first time of use. In subsequent times, you can just select your profile.

    3. Open the profile data file.

    After login, click on LOAD PROFILE DATA FILE. This opens a file browser of your home directory on the OSC cluster you logged onto. Go to the directory that contains the .map file and select it. This will open the file and allow you to navigate the source code line-by-line and investigate the performance characteristics. 

    A license is not required to simply open the client, so it is possible to skip 2. Configure the client, if you download the profile data file to your desktop. You can then open it by just selecting LOAD PROFILE DATA FILE and navigating through a file browser on your local system.

    Note that the client is Allinea Forge, a client that contains Allinea MAP and Allinea DDT. Allinea DDT is a debugger, and OSC has license only for Allinea MAP. If you need a debugger, you can use Totalview instead.

    See Also

    Documentation Attachment: 
    Service: 
    Fields of Science: 

    Altair HyperWorks

    HyperWorks is a high-performance, comprehensive toolbox of CAE software for engineering design and simulation.

    Availability & Restrictions

    HyperWorks is available to all academic clients.

    The following version of Altair Hyperworks can be found for the following environments:

    Version Oakley Statewide
    10.0   X
    11.0 X X
    12.0 X X
    13.0 X X
    14.0 X X
    2017.1 X X

     

    Please contact OSC Help to request the appropriate form for access.

    Usage

    Using HyperWorks through OSC installation

    To use HyperWorks on the OSC clusters, first ensure that X11 forwarding is enabled as the HyperWorks workbench is a graphical application. Then, load the hyperworks module:

    module load hyperworks
    

    The HyperWorks GUI can be launched then with the following command:

    hw
    

    State-wide access for HyperWorks

    For information on downloading and installing a local copy through the state-wide license, follow the steps below. 

    NOTE: To run Altair HyperWorks, your computer must have access to the internet. The software contacts the license server at OSC to check out a license when it starts and periodically during execution. The amount of data transferred is small, so network connections over modems are acceptable.

     

    Usage of HyperWorks on a local machine using the statewide license will vary from installation to installation.

    1. Go to http://www.altairhyperworks.com/ 

    2. Click on "Login" in the upper right hand corner of the page. 

    3. If you have already registered with the Altair web site, enter the e-mail address that you registered with and your password and skip to step #5.

    4. If you have not registered yet, click the link that says "Click here to register now". You will be prompted for some contact information and an e-mail address which will be your unique identifier.

      • IMPORTANT: The e-mail address you give must be from your academic institution. Under the statewide license agreement, registration from Ohio universities is allowed on the Altair web site. Trying to log in with a yahoo or hotmail e-mail account will not work. If you enter your university e-mail and the system will not register you, please contact OSChelp at oschelp@osc.edu.

    5. Once you have logged in, click on "SUPPORT" and then "SOFTWARE DOWNLOADS"

    6. In addition to downloading the software, download the "Installation Guide and Release Notes" for instructions on how to install the software.

      • IMPORTANT: If you have any questions or problems, please contact OSChelp at oschelp@osc.edu, rather than HyperWorks support. The software agreements outlines that problems should first be sent to OSC. If the OSC support line cannot answer or resolve the question, they have the ability to raise the problem to Altair support.

    7. Please contact OSC Help for further instruction and license server information. In order to be added to the allowed list for the state-wide software access, we will need your IP address/range of machine that will be running this software. 

    8. You need to set an environment variable (ALTAIR_LICENSE_PATH) on your local machine to point at our license server (7790@license6.osc.edu). See this link for instructions if necessary.

    Further Reading

    For more information about HyperWorks, see the following:

    See Also

    Service: 
    Fields of Science: 

    BLAS

    The BLAS (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations.

    Availability and Restrictions

    A highly optimized implementation of the BLAS is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building the BLAS for yourself. MKL is available to all OSC users.

    Usage

    See OSC's MKL software page for usage information. Note that there is no library named libblas.a or libblas.so. The flag "-lblas" on your link line will not work. You should modify your makefile or build script to link to the MKL libraries instead.

    Further Reading

    Service: 
    Technologies: 
    Fields of Science: 

    BLAST

    The BLAST programs are widely used tools for searching DNA and protein databases for sequence similarity to identify homologs to a query sequence. While often referred to as just "BLAST", this can really be thought of as a set of programs: blastp, blastn, blastx, tblastn, and tblastx.

    Availability & Restrictions

    BLAST is available without restriction to all OSC users.

    The following versions of BLAST are available on OSC systems:

    Version Oakley Owens
    2.2.24+ X  
    2.2.25+ X  
    2.2.26 X  
    2.2.31+ X  
    2.4.0+   X*
    2.6.0+ X*  

     

    If you need to use blastx, you will need to load one of the C++ implimenations modules of blast (any version with a "+").

    Set-up

    To load BLAST, type the following into the command line:

    module load blast
    

    Then create a resource file .ncbirc, and put it under your home directory.

    Using BLAST

    The five flavors of BLAST mentioned above perform the following tasks:

    blastp: compares an amino acid query sequence against a protein sequence database
    
    blastn: compares a nucleotide query sequence against a nucleotide sequence database
    
    blastx: compares the six-frame conceptual translation products of a nucleotide query sequence (both strands) against a protein sequence database
    
    tblastn: compares a protein query sequence against a nucleotide sequence database dynamically translated in all six reading frames (both strands).
    
    tblastx: compares the six-frame translations of a nucleotide query sequence against the six-frame translations of a nucleotide sequence database. (Due to the nature of tblastx, gapped alignments are not available with this option)
    

    We provide local access to nt and refseq_protein databases. You can access the database using BLASTDB environment variable. If you need other databases, please send a request email to OSC Help .

    Batch Usage

    A sample batch script is below:

    #PBS -l nodes=1:ppn=1
    #PBS -l walltime=10:00
    #PBS -N Blast
    #PBS -S /bin/bash
    #PBS -j oe
    module load blast
    set -x
    cd $PBS_O_WORKDIR
    mkdir $PBS_JOBID
    cp 100.fasta $TMPDIR
    cd $TMPDIR
    /usr/bin/time blastn -db nt -query 100.fasta -out test.out
    cp test.out $PBS_O_WORKDIR/$PBS_JOBID
    

    NCBI BLAST Database

    The NCBI BLAST database is updated on Oct 30th 2017 and is available  at /fs/project/pub_data/blastDB-Oct2017/db  

    Further Reading

    Service: 
    Fields of Science: 

    BLAT

    BLAT is a sequence analysis tool which performs rapid mRNA/DNA and cross-species protein alignments. BLAT is more accurate and 500 times faster than popular existing tools for mRNA/DNA alignments and 50 times faster for protein alignments at sensitivity settings typically used when comparing vertebrate sequences.

    BLAT is not BLAST. DNA BLAT works by keeping an index of the entire genome (but not the genome itself) in memory. Since the index takes up a bit less than a gigabyte of RAM, BLAT can deliver high performance on a reasonably priced Linux box. The index is used to find areas of probable homology, which are then loaded into memory for a detailed alignment. Protein BLAT works in a similar manner, except with 4-mers rather than 11-mers. The protein index takes a little more than 2 gigabytes.

    Availability & Restrictions

    BLAT is available without restriction to all OSC users.

    The following versions of BLAT are available at OSC:

    Version Oakley
    34 X

    Usage

    Set-up

    To initalize the Oakley system prior to using BLAT, run the following commands:

    module load blat

    Using BLAT

    The main programs in the blat suite are:

    gfServer – a server that maintains an index of the genome in memory and uses the index to quickly find regions with high levels of sequence similarity to a query sequence.
    gfClient – a program that queries gfServer over the network, and then does a detailed alignment of the query sequence with regions found by gfServer.
    blat –combines client and server into a single program, first building the index, then using the index, and then exiting. 
    webBlat – a web based version of gfClient that presents the alignments in an interactive fashion. (not included on OSC server)

    Building an index of the genome typically takes 10 or 15 minutes.  Typically for interactive applications one uses gfServer to build a whole genome index.  At that point gfClient or webBlat can align a single query within few seconds.  If one is aligning a lot of sequences in a batch mode then blat can be more efficient, particularly if run on a cluster of computers.  Each blat run is typically done against a single chromosome, but with a large number of query sequences.

    Other programs in the blat suite are:

    pslSort – combines and sorts the output of multiple blat runs.  (The blat default output format is .psl).
    pslReps – selects the best alignments for a particular query sequence, using a ‘near best in genome’ approach.
    pslPretty – converts alignments from the psl format, which is tab-delimited format and does not include the bases themselves, to a more readable alignment format.
    faToTwoBit – convert Fasta format sequence files to a dense randomly-accessable  .2bit format that gfClient can use.
    twoBitToFa – convert from the .2bit format back to fasta
    faToNib – convert from Fasta to a somewhat less dense randomly accessible format that predates .2bit.  Note each .nib file can only contain a single sequence.
    nibFrag – convert portions of a nib file back to fasta.

    The command line options of each of the programs is described below. Similar summaries of usage are printed when a command is run with no arguments.

    Batch Usage

    A sample batch script is as below:

    #PBS -N blat
    #PBS -j oe
    #PBS -l nodes=1:ppn=1
    #PBS -S /bin/bash
    
    module load blat 
    
    cd $PBS_O_WORKDIR
    cp $BLAT_TEST_DIR/gfServer-tests/input/creaGeno.2bit .
    cp $BLAT_TEST_DIR/gfServer-tests/input/crea.mrna .
    
    blat -stepSize=5 -repMatch=2253 -minScore=0 -minIdentity=0 creaGeno.2bit crea.mrna output.psl 

     

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    BWA

    BWA is a software package for mapping low-divergent sequences against a large reference genome, such as the human genome. It consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM.

    Availability and Restrictions

    The following versions of BWA are available on OSC clusters:

    Version Oakley owens
    0.7.13 X* X*
    *: Current default version

    You can use  module spider bwa to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    BWA is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of BWA, run the following command:   module load bwa. The default version will be loaded. To select a particular BWA version, use   module load bwa/version. For example, use  module load bwa/0.7.13 to load BWA 0.7.13.

    Usage on Owens

    Set-up

    To configure your environment for use of BWA, run the following command:   module load bwa. The default version will be loaded. To select a particular BWA version, use   module load bwa/version. For example, use module load bwa/0.7.13 to load BWA 0.7.13.

    Further Reading

    Service: 
    Fields of Science: 

    BamTools

    BamTools provides both a programmer's API and an end-user's toolkit for handling BAM files.

    Availability and Restrictions

    The following versions of BamTools are available on OSC clusters:

    Version Oakley Owens
    2.2.2 X X*
    2.4.0 X*  
    *: Current default version

    You can use  module spider bamtools to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    BamTools is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of BamTools, run the following command:   module load bamtools . The default version will be loaded. To select a particular BamTools version, use  module load bamtools/version . For example, use  module load bamtools/2.2.2 to load BamTools 2.2.2.

    Usage on Owens

    Set-up

    To configure your environment for use of BamTools, run the following command:   module load bamtools . The default version will be loaded. To select a particular BamTools version, use  module load bamtools/version . For example, use  module load bamtools/2.2.2 to load BamTools 2.2.2.

    Further Reading

    Service: 
    Fields of Science: 

    BioPerl

    BioPerl offers a set of perl modules which can be used for sequence manipulation. Knowledge of PERL programming is required.

    Availability & Restrictions

    BioPerl is available without restriction to all OSC users.

    The following versions of BioPerl are available:

    Version Oakley
    1.6.1 X

    Usage

    Using BioPerl

    This is an example of how to use bioperl and access the sequence package on the Oakley Cluster.

    use lib '/usr/local/bioperl/1.6.1/lib/perl5/'
    use Bio::Seq;

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    Boost

    Boost is a set of C++ libraries that provide helpful data structures and numerous support functions in a wide range of aspects of programming, such as, image processing, gpu programming, concurrent programming, along with many algorithms.  Boost is portable and performs well on a wide variety of platforms.

    Availability & Restrictions

    Boost is available without restriction to all OSC Users

    The following version of Boost are available on OSC systems:

    Version Oakley Ruby owens Notes
    1.41.0 System Install System Install   No Module Needed
    1.53.0     System Install No Module Needed
    1.56.0 X(GI) X(GI)    
    1.58.0 X(G)      
    1.60.0 X(G)      
    1.63.0  X(GI)*  X(GI)*  X(GI)*  
    1.64.0 X(GI) X(GI) X(GI)  

     

    NOTE: G = available with gnu; I = available with intel; * means default version

    Usage on Oakley

    Set-up

    Initalizing the system for use of the Boost library is independent of the compiler you are using.  To load the boost module run the following command:

    module load boost
    

    Building With Boost

    The following environment variables are setup when the Boost library is loaded:

    Variable Use
    $BOOST_CFLAGS Use during your compilation step for C++ programs.
    $BOOST_LIBS Use during your link step.

     

    Below is a set of example commands used to build and run a file called example2.cpp. First copy the example2.cpp and jayne.txt with the following commands.

    cp /usr/local/src/boost/boost-1.56.0/test.osc/example2.cpp .
    cp /usr/local/src/boost/boost-1.56.0/test.osc/jayne.txt .
    module load boost
    g++ $BOOST_CFLAGS example2.cpp -o boostTest $BOOST_LIBS -lboost_regex
    ./boostTest < jayne.txt
    

    Usage on Ruby

    Set-up

    Initalizing the system for use of the Boost library is independent of the compiler you are using.  To load the boost module run the following command:

    module load boost
    

    Building With Boost

    The following environment variables are setup when the Boost library is loaded:

    VARIABLE USE
    $BOOST_CFLAGS Use during your compilation step for C++ programs.
    $BOOST_LIBS Use during your link step.

     

    Below is a set of example commands used to build and run a file called  example2.cpp. First copy the example2.cpp and jayne.txt from Oakley into your home directory with the following commands:

    cp /usr/local/src/boost/boost-1_56_0/test.osc/example2.cpp ~
    cp /usr/local/src/boost/boost-1_56_0/test.osc/jayne.txt ~
    
    Then compile and test the program with the folllowing commands:
    module load boost
    g++ $BOOST_CFLAGS example2.cpp -o boostTest $BOOST_LIBS -lboost_regex
    ./boostTest < jayne.txt
    

    Usage on Owens

    Set-up

    Since Boost is a System Install, no module is needed to run the application.

    Building With Boost

    Below is a set of example commands used to build and run a file called  example2.cpp. First copy the example2.cpp and jayne.txt from Oakley into your home directory with the following commands:

    cp /usr/local/src/boost/boost-1_56_0/test.osc/example2.cpp ~
    cp /usr/local/src/boost/boost-1_56_0/test.osc/jayne.txt ~
    
    Then compile and test the program with the folllowing commands:
    g++ example2.cpp -o boostTest -lboost_regex
    ./boostTest < jayne.txt
    

    Further Reading

     

    Service: 
    Fields of Science: 

    Bowtie1

    Bowtie1 is an ultrafast, memory-efficient short read aligner. It aligns short DNA sequences (reads) to the human genome at a rate of over 25 million 35-bp reads per hour. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small: typically about 2.2 GB for the human genome (2.9 GB for paired-end).

    Availability and Restrictions

    The following versions of Bowtie1 are available on OSC clusters:

    Version Oakley Owens
    1.1.2 X* X*
    *: Current default version

    You can use  module spider bowtie1   to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Bowtie1 is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of Bowtie1, run the following command:   module load bowtie1 . The default version will be loaded. To select a particular Bowtie1 version, use   module load bowtie/version . For example, use  module load bowtie1/1.1.2 to load Bowtie1 1.1.2.

    Usage on Owens

    Set-up

    To configure your environment for use of Bowtie1, run the following command:    module load bowtie1 . The default version will be loaded. To select a particular Bowtie1 version, use    module load bowtie/version . For example, use  module load bowtie1/1.1.2 to load Bowtie1 1.1.2.

    Further Reading

    Service: 
    Fields of Science: 

    Bowtie2

    Bowtie 2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. It is particularly good at aligning reads of about 50 up to 100s or 1,000s of characters, and particularly good at aligning to relatively long (e.g. mammalian) genomes. Bowtie 2 indexes the genome with an FM Index to keep its memory footprint small: for the human genome, its memory footprint is typically around 3.2 GB. Bowtie 2 supports gapped, local, and paired-end alignment modes.

    Availability and Restrictions

    The following versions of Bowtie2 are available on OSC clusters:

    Version Oakley Owens
    2.2.8 X*  
    2.2.9   X*
    *: Current default version

    You can use  module spider bowtie2  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Bowtie2 is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of Bowtie2, run the following command:   module load bowtie2  . The default version will be loaded. To select a particular Bowtie2 version, use   module load bowtie2/version . For example, use  module load bowtie2/2.2.8  to load Bowtie2 2.2.8.

    Usage on Owens

    Set-up

    To configure your environment for use of Bowtie2, run the following command:    module load bowtie2  . The default version will be loaded. To select a particular Bowtie2 version, use    module load bowtie2/version . For example, use  module load bowtie2/2.2.9  to load Bowtie2 2.2.9.

    Further Reading

    Service: 
    Fields of Science: 

    CDO

    CDO (Climate Data Operators) is a collection of command line operator to manipulate and analyse climate and NWP model data. It is open source and released under the terms of the GNU General Public License v2 (GPL).

    Availability and Restrictions

    Versions

    CDO is available on Oakley Cluster. The versions currently available at OSC are:

    version oakley
    1.5.4 X
    1.6.8 X*
    * : Default Version

    You can use module spider cdo  to view available modules on Oakley. Feel free to contact OSC Help if you need other versions for your work.

    Access 

    CDO is available for use by all OSC users.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version of CDO module, use   module load cdo . To select a particular software version, use   module load cdo/version . For example, use   module load cdo/1.5.4  to load CDO version 1.5.4 on Oakley. 

    Batch Usauge on Oakley

    Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for Oakley and Scheduling Policies and Limits for more info. 
    Interactive Batch Session
    For an interactive batch session on Oakley, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Once your job is ready, you can start to use CDO. For example, use the following command to display general information about a data file named infile :
    cdo info infile
    

    Use cdo -h [operator] for operator descriptions. 

    Further Reading

    Official documentation can be obtained from https://code.zmaw.de/projects/cdo/wiki/Cdo#Documentation

    Service: 
    Fields of Science: 

    COMSOL

    COMSOL Multiphysics (formerly FEMLAB) is a finite element analysis and solver software package for various physics and engineering applications, especially coupled phenomena, or multiphysics. owned and supported by COMSOL, Inc.

    Availability and Restrictions

    COMSOL is available on the Oakley and Ruby clusters. The versions currently available at OSC are

    Version oakley Ruby Owens notes
    4.3 X      
    4.3a X     Default version on Oakley prior to 09/15/2015
    4.3b X      
    4.4 X      
    5.1 X      
    5.2 X X*    
    5.2a X*   X*  
    5.3 X      
    *: Current default version

    You can use module spider comsol  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    COMSOL is for academic use, available to all Ohio State University users. OSC does not provide COMSOL licenses for academic use to students and faculty outside of Ohio State University due to licensing restrictions. If you or your institution have a network COMSOL license server, you may be able to use it on OSC. For connections to your license server from OSC, please read this document. If you need further help, please contact OSC Help .

    To use COMSOL you will have to be added to the license server first.  Please contact OSC Help to be added.

    Access for Commerical Users

    Contact OSC Help for getting access to COMSOL if you are a commercial user. 

    Usage on Oakley

    Set-up

    To load the default version of COMSOL module, use  module load comsol . To select a particular software version, use   module load comsol/version . For example, use  module load comsol/44  to load COMSOL version 4.4. 

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00 -l software=comsolscript
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    Assume that you have had a comsol script file mycomsol.m  in your working direcory ( $PBS_O_WORKDIR ). Below is the example batch script ( job.txt ) for a serial run: 

    #PBS -N COMSOL
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    #PBS -l software=comsolscript
    #
    # The following lines set up the COMSOL environment
    #
    module load comsol
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.m $TMPDIR
    cd $TMPDIR
    #
    # Run COMSOL
    #
    comsol batch mycomsol
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    
    Non-interactive Batch Job (Parallel Run for COMSOL 4.3 and Later)

    As of version 4.3, it is not necessary to start up MPD before launching a COMSOL job. Below is the example batch script ( job.txt ) for a parallel run using COMSOL 4.3 or later versions:

    #PBS -l walltime=01:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -N COMSOL
    #PBS -j oe
    #PBS -r n
    #PBS -l software=comsolscript
    cd ${PBS_O_WORKDIR}
    module load comsol
    echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
    cp input_cluster.mph $TMPDIR
    cd $TMPDIR
    np=12
    echo "--- Running on ${np} processes (cores) on the following nodes:"
    cat $PBS_NODEFILE | uniq > hostfile
    echo "--- COMSOL run"
    comsol -nn 2 -np ${np} batch -f hostfile -mpirsh rsh -inputfile input_cluster.mph -outputfile output_cluster.mph
    echo "--- Copy files back"
    cp output_cluster.mph output_cluster.mph.status ${PBS_O_WORKDIR}
    echo "---Job finished at: 'date'"
    echo "---------------------------------------------"
    

    Note:

    • Set nodes to 2 and ppn to 12 ( -l nodes=2:ppn=12  ). You can change the values per your need.
    • Copy files from your directory to $TMPDIR.
    • Provide the name of the input file and output file.

    Usage on Ruby

    Set-up

    To load the default version of COMSOL module, use   module load comsol  . To select a particular software version, use    module load comsol/version  . For example, use   module load comsol/52   to load COMSOL version 5.2. 

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00 -l software=comsolscript
    
    which gives you 20 cores (  -l nodes=1:ppn=20  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    Assume that you have had a comsol script file  mycomsol.m   in your working direcory (  $PBS_O_WORKDIR  ). Below is the example batch script (  job.txt  ) for a serial run: 

    #PBS -N COMSOL
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    #PBS -l software=comsolscript
    #
    # The following lines set up the COMSOL environment
    #
    module load comsol
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.m $TMPDIR
    cd $TMPDIR
    #
    # Run COMSOL
    #
    comsol batch mycomsol
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    
    Non-interactive Batch Job (Parallel Run for COMSOL 4.3 and Later)

    As of version 4.3, it is not necessary to start up MPD before launching a COMSOL job. Below is the example batch script (  job.txt  ) for a parallel run using COMSOL 4.3 or later versions:

    #PBS -l walltime=01:00:00
    #PBS -l nodes=2:ppn=20
    #PBS -N COMSOL
    #PBS -j oe
    #PBS -r n
    #PBS -l software=comsolscript
    cd ${PBS_O_WORKDIR}
    module load comsol
    echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
    cp input_cluster.mph $TMPDIR
    cd $TMPDIR
    np=20
    echo "--- Running on ${np} processes (cores) on the following nodes:"
    cat $PBS_NODEFILE | uniq > hostfile
    echo "--- COMSOL run"
    comsol -nn 2 -np ${np} batch -f hostfile -mpirsh rsh -inputfile input_cluster.mph -outputfile output_cluster.mph
    echo "--- Copy files back"
    cp output_cluster.mph output_cluster.mph.status ${PBS_O_WORKDIR}
    echo "---Job finished at: 'date'"
    echo "---------------------------------------------"
    

    Note:

    • Set nodes to 2 and ppn to 20 (  -l nodes=2:ppn=20 ). You can change the values per your need.
    • Copy files from your directory to $TMPDIR.
    • Provide the name of the input file and output file.

    Usage on Owens

    Set-up

    To load the default version of COMSOL module, use   module load comsol  . To select a particular software version, use    module load comsol/version  . For example, use   module load comsol/52a   to load COMSOL version 5.2a. 

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00 -l software=comsolscript
    
    which gives you 28 cores (  -l nodes=1:ppn=28  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    Assume that you have had a comsol script file  mycomsol.m   in your working direcory (  $PBS_O_WORKDIR  ). Below is the example batch script (  job.txt  ) for a serial run: 

    #PBS -N COMSOL
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    #PBS -l software=comsolscript
    #
    # The following lines set up the COMSOL environment
    #
    module load comsol
    #
    # Move to the directory where the job was submitted
    #
    cd $PBS_O_WORKDIR
    cp *.m $TMPDIR
    cd $TMPDIR
    #
    # Run COMSOL
    #
    comsol batch mycomsol
    #
    # Now, copy data (or move) back once the simulation has completed
    #
    cp * $PBS_O_WORKDIR
    
    Non-interactive Batch Job (Parallel Run for COMSOL 4.3 and Later)

    As of version 4.3, it is not necessary to start up MPD before launching a COMSOL job. Below is the example batch script (  job.txt  ) for a parallel run using COMSOL 4.3 or later versions:

    #PBS -l walltime=01:00:00
    #PBS -l nodes=2:ppn=28
    #PBS -N COMSOL
    #PBS -j oe
    #PBS -r n
    #PBS -l software=comsolscript
    cd ${PBS_O_WORKDIR}
    module load comsol
    echo "--- Copy Input Files to TMPDIR and Change Disk to TMPDIR"
    cp input_cluster.mph $TMPDIR
    cd $TMPDIR
    np=28
    echo "--- Running on ${np} processes (cores) on the following nodes:"
    cat $PBS_NODEFILE | uniq > hostfile
    echo "--- COMSOL run"
    comsol -nn 2 -np ${np} batch -f hostfile -mpirsh rsh -inputfile input_cluster.mph -outputfile output_cluster.mph
    echo "--- Copy files back"
    cp output_cluster.mph output_cluster.mph.status ${PBS_O_WORKDIR}
    echo "---Job finished at: 'date'"
    echo "---------------------------------------------"
    

    Note:

    • Set nodes to 2 and ppn to 28 (  -l nodes=2:ppn=28   ). You can change the values per your need.
    • Copy files from your directory to $TMPDIR.
    • Provide the name of the input file and output file.

    Further Reading

    Service: 

    Interactive Parallel COMSOL Job

    This documentation is to discuss how to set up an interactive parallel COMSOL job at OSC. The following example demonstrates the process of using COMSOL version 5.1 on Oakley. Depending on the version of COMSOL and cluster you work on, there mighe be some differences from the example. Feel free to contact OSC Help if you have any questions. 

    • Launch COMSOL GUI application following the instructions on this page. Get the information on the node(s) allocated to your job and save it in the file named hostfile using the following command:

     

    cat $PBS_NODEFILE | uniq > hostfile
    

    Make sure the hostfile is located in the same directory where you COMSOL input file is put

    • Open COMSOL GUI application. To enable the cluster compuitng feature, click the show button and select Advanced Study Options, as shown in the picture below:

    Advanced study

    • In the Model Builder, right-click a Study node and select Cluster Computing, as shown in the picture below:

    cluster computing

    • In the Cluster Computing node's setting window, select General from the Cluster type list. Provide the name of Host file as hostfile. Browse to the directory where your COMSOL input file is located, as shown in the picture below:

    setting

    • Save all the settings. Then you should be able to run an interactive parallel COMSOL job at OSC

    CUDA

    CUDA™ (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by Nvidia that enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU).

    Availability and Restrictions

    CUDA is available on the clusters supporting GPUs. The versions currently available at OSC are

    Version Oakley Ruby Owens notes
    5.0.35 X X   Default version on Oakley prior to 09/15/2015
    5.5 X X    
    6.0.37 X X    
    6.5.14 X* X    
    7.0.28 X X    
    7.5.18 X X    

    8.0.44

    8.0.61

     

     

    X*

    X

    X*

    GPU drivers prevent the use of CUDA 8 on Oakley

    *: Current default version

    You can use module spider cuda to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    CUDA is available for use by all OSC users.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version of CUDA module, use  module load cuda . To select a particular software version, use   module load cuda/version . For example, use  module load cuda/7.0.28 to load CUDA version 7.0.28 on Oakley. 

    GPU Computing SDK

    The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. On Oakley, the SDK binaries are located in $CUDA_HOME/bin/linux/release ( $CUDA_HOME is an environment variable set when you load the module).

    Programming in CUDA

    Please visit the following link to learn programming in CUDA, http://developer.nvidia.com/cuda-education-training. The link also contains tutorials on Optimizing CUDA codes to obtain greater SpeedUp. One can also refer to the following webpage for some more CUDA optimization techniques, http://www.cs.berkeley.edu/~volkov/

    Compiling CUDA Code

    One can type module show cuda-version-number to view the list of environment variables.
    To compile a CUDA C code contained in a file, let say mycudaApp.cu , the following could be done after loading the appropriate CUDA module: nvcc -o mycudaApp mycudaApp.cu This will create an executable by name mycudaApp .

    Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

    Debugging CUDA code

    cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

    Detecting memory access errors

    CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=1:gpus=1 -l walltime=00:20:00
    

    which requests one core ( -l nodes=1:ppn=1 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Please note that on Oakley, you can request any mix of ppn and gpus you need; please see the Batch Limit Rules and Job Scripts page in our batch guide for more information.
    Non-interactive Batch Job (Serial Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (  job.txt ) for a serial run:

    #PBS -l walltime=01:00:00
    #PBS -l nodes=1:ppn=1:gpus=1
    #PBS -N compute
    #PBS -j oe
    module load cuda
    cd $HOME/cuda
    cp mycudaApp $TMPDIR
    cd $TMPDIR
    ./mycudaApp
    

    Usage on Ruby

    Set-up on Ruby

    To load the default version of CUDA module, use  module load cuda . To select a particular software version, use   module load cuda/version . For example, use  module load cuda/7.0.28 to load CUDA version 7.0.28 on Ruby. 

    GPU Computing SDK

    The NVIDIA GPU Computing SDK provides hundreds of code samples and covers a wide range of applications/techniques to help you get started on the path of writing software with CUDA C/C++ or DirectCompute. 

    Programming in CUDA

    Please visit the following link to learn programming in CUDA, http://developer.nvidia.com/cuda-education-training. The link also contains tutorials on Optimizing CUDA codes to obtain greater SpeedUp. One can also refer to the following webpage for some more CUDA optimization techniques, http://www.cs.berkeley.edu/~volkov/

    Compiling CUDA Code

    One can type module show cuda-version-number to view the list of environment variables.
    To compile a cuda code contained in a file, let say mycudaApp.cu , the following could be done after loading the appropriate CUDA module: nvcc -o mycudaApp mycudaApp.cu This will create an executable by name mycudaApp .

    Important: The devices are configured in exclusive mode. This means that 'cudaSetDevice' should NOT be used if requesting one GPU resource. Once the first call to CUDA is executed, the system will figure out which device it is using. If both cards per node is in use by a single application, please use 'cudaSetDevice'.

    Debugging CUDA code

    cuda-gdb can be used to debug CUDA codes. module load cuda will make it available to you. For more information on how to use the CUDA-GDB please visit http://developer.nvidia.com/cuda-gdb.

    Detecting memory access errors

    CUDA-MEMCHECK could be used for detecting the source and cause of memory access errors in your program. For more information on how to use CUDA-MEMCHECK please visit http://developer.nvidia.com/cuda-memcheck.

    Setting the GPU compute mode on Ruby

    The GPUs on Ruby can be set to different compute modes as listed here.

    They can be set by adding the following to the GPU specification:

    -l nodes=1:ppn=20:gpus=1:default
    -l nodes=1:ppn=20:gpus=1:exclusive
    -l nodes=1:ppn=20:gpus=1:exclusive_process
    

    Note, the prohibited mode is not an option.

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=20:gpus=1 -l walltime=00:20:00 
    

    which requests one whole node with 20 cores ( -l nodes=1:ppn=20 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Non-interactive Batch Job (Serial Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script (  job.txt ) for a serial run:

    #PBS -l walltime=01:00:00
    #PBS -l nodes=1:ppn=20:gpus=1
    #PBS -N compute
    #PBS -j oe
    module load cuda
    cd $HOME/cuda
    cp mycudaApp $TMPDIR
    cd $TMPDIR
    ./mycudaApp
    

    Further Reading

    Online documentation is available at http://developer.nvidia.com/nvidia-gpu-computing-documentation

    See Also

    Service: 
    Technologies: 
    Fields of Science: 

    Caffe

    Caffe is "a fast open framework for deep learning".

    From their README:

    Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by community contributors. Yangqing Jia created the project during his PhD at UC Berkeley. Caffe is released under the BSD 2-Clause license.
    

    Caffe also includes interfaces for both Python and Matlab, which have been built but have not been tested.

    Availability and Restrictions

    Versions

    The following versions of Caffe are available on OSC clusters:

    version Ruby Owens
    1.0.0-rc3 X X

     

    The current versions of Caffe on Ruby require the use of python/2.7.8, cuda/7.5.18, boost/1.56.0, opencv/3.1.0 on Ruby for GPU calculations.

    The current version of Caffe on Owens requires cuda/8.0.44 for GPU calculations.

    Feel free to contact OSC Help if you need other versions for your work.

    Access 

    Caffe is available to all OSC users without restriction.

    Usage

    Usage on Ruby or Owens

    Setup on Ruby or Owens

    To configure the Ruby or Owens cluster for the use of Caffe, use the following commands:

    module load caffe
    

    Batch Usage on Ruby or Owens

    Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for Ruby, and Scheduling Policies and Limits for more info.  In particular, Caffe should be run on a GPU-enabled compute node.

    An Example of Using Caffe with MNIST Training Data on Ruby

    Below is an example batch script ( job.txt ) for using Caffe, see this resource for a detailed explanation http://caffe.berkeleyvision.org/gathered/examples/mnist.html

    #PBS -N Caffe
    #PBS -l nodes=1:ppn=20:gpus=1:default
    #PBS -l walltime=30:00
    #PBS -j oe
    #PBS -S /bin/bash 
    cd $PBS_O_WORKDIR
    . /etc/profile.d/lmod.sh
    # Load the modules for Caffe
    ml caffe
    # Migrate to job temp directory and copy folders over
    cd $TMPDIR
    cp -r $CAFFE_HOME/{examples,build,data} .
    # Download, create, train
    ./data/mnist/get_mnist.sh
    ./examples/mnist/create_mnist.sh
    ./examples/mnist/train_lenet.sh
    # Serialize log files
    echo; echo 'LOG 1'
    cat convert_mnist_data.bin.$(hostname)*
    echo; echo 'LOG 2'
    cat caffe.INFO
    echo; echo 'LOG 3'
    cat convert_mnist_data.bin.INFO
    echo; echo 'LOG 4'
    cat caffe.$(hostname).*
    cp examples/mnist/lenet_iter_10000.* $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the following command:

    qsub job.txt
    
    An Example of Using Caffe with MNIST Training Data on Owens

    Below is an example batch script ( job.txt) for using Caffe, see this resource for a detailed explanation http://caffe.berkeleyvision.org/gathered/examples/mnist.html

    #!/bin/bash
    #PBS -N Caffe
    #PBS -l nodes=1:ppn=28:gpu
    #PBS -l walltime=30:00
    #PBS -j oe
    #PBS -S /bin/bash 
    cd $PBS_O_WORKDIR
    . /etc/profile.d/lmod.sh
    # Load the modules for Caffe
    ml caffe
    # Migrate to job temp directory and copy folders over
    cd $TMPDIR
    cp -r $CAFFE_HOME/{examples,data} .
    # Download, create, train
    ./data/mnist/get_mnist.sh
    ./examples/mnist/create_mnist.sh
    ./examples/mnist/train_lenet.sh
    # Serialize log files
    echo; echo 'LOG 1'
    cat convert_mnist_data.bin.$(hostname)*
    echo; echo 'LOG 2'
    cat caffe.INFO
    echo; echo 'LOG 3'
    cat convert_mnist_data.bin.INFO
    echo; echo 'LOG 4'
    cat caffe.$(hostname).*
    cp examples/mnist/lenet_iter_10000.* $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the following command:

    qsub job.txt
    

    Further Reading

    http://caffe.berkeleyvision.org/

    Service: 
    Technologies: 
    Fields of Science: 

    Cairo

    Cairo is a 2D graphics library with support for multiple output devices. It is designed to produce consistent output on all output media while taking advantage of display hardware acceleration when available.

    Availability and Restrictions

    Versions

    The following versions of Cairo are available on OSC clusters:

    VERSION OAKLEY Owens
    1.12.18 X(GI)  
    1.14.2 X(GI)* System Install(GI)*
    • X(GI): available with gnu and intel
    • *: Default Versio

    You can use  module spider cairo  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Cairo is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of Cairo, run the following command:  module load cairo . The default version will be loaded. To select a particular Cairo version, use  module load cairo/version . For example, use module load cairo/1.14.2 to load Cairo 1.14.2.

    Building with Cairo

    The Cairo library provides the following variables for use at build time:

    Variable Use
    $CAIRO_CFLAGS Use during your compilation step for programs
    $CAIRO_LIBS Use during your linking step programs

     

    For example, to build the code myprog.c with the Cairo library you would use:

    icc -o myprog myprog.c $CAIRO_LIBS $CAIRO_CFLAGS
    

    Usage on Owens

    Set-up

    To configure your environment for use of Cairo, run the following command:  module load cairo . The default version will be loaded. To select a particular Cairo version, use  module load cairo/version . For example, use module load cairo/1.14.2 to load Cairo 1.14.2.

    Building with Cairo

    The Cairo library provides the following variables for use at build time:

    VARIABLE USE
    $CAIRO_CFLAGS Use during your compilation step for programs
    $CAIRO_LIBS Use during your linking step programs (Note: Dynamic linking only)

     

    For example, to build the code myprog.c with the Cairo library you would use:

    icc -o myprog myprog.c -lcairo

    Further Reading

    Service: 
    Fields of Science: 

    Cambridge Structural Database

    Introduction

    The Cambridge Structural Database (CSD) contains complete structure information on hundreds of thousands of small molecule crystals. The Cambridge Crystallographic Data Centre (CCDC) is a suite of programs for CSD search and analysis. It includes the graphical user interface (GUI) programs ConQuest or searching and retrieving information from the CSD and Mercury for structural visualization and analysis. The Cambridge Crystallographic Data Centre Home Page has additional information.

    Version

    CSD 2017 is available only for academic OSU users.

    There are also a limited number of licenses for the web version WebCSD.

    Before you can use CSD we need to add you to our system. Please contact OSC Help for availability.

    Availability

    CSD is available on Oakley login nodes.  For the GUI applications, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

    Usage

    CSD is only available to The Ohio State University academic users. In order to access this software license, please contact OSC Help for further instruction.

    To run conquest:

    module load csd
    cq
    

    Documentation

    General documentation is available at the CCDC Documentation page and in the local machine directories.

    Service: 

    Clustal W

    Clustal W is a general purpose multiple sequence alignment program for DNA or proteins.It produces biologically meaningful multiple sequence alignments of divergent sequences. It calculates the best match for the selected sequences, and lines them up so that the identities, similarities and differences can be seen.

    Availability & Restrictions

    Clustal W is available without restriction to all OSC users.

    The following versions of Clustal W are available on OSC systems:

    Version Oakley
    2.1 X

    Usage

    Set-up

    On the Oakley system load the clustalw module:

    module load clustalw

     

    Using Clustal W

    Once the clustalw module has been loaded, the commands are available for your use. On the Oakley system, the command is

    clustalw2

    The options can be listed interactively by typing clustalw -help or clustalw -check on the command-line.

                    DATA (sequences)
    -INFILE=file.ext                             :input sequences.
    -PROFILE1=file.ext  and  -PROFILE2=file.ext  :profiles (old alignment).
    
                    VERBS (do things)
    -OPTIONS            :list the command line parameters
    -HELP  or -CHECK    :outline the command line params.
    -ALIGN              :do full multiple alignment 
    -TREE               :calculate NJ tree.
    -BOOTSTRAP(=n)      :bootstrap a NJ tree (n= number of bootstraps; def. = 1000).
    -CONVERT            :output the input sequences in a different file format.
    
                    PARAMETERS (set things)
    ***General settings:****
    -INTERACTIVE :read command line, then enter normal interactive menus
    -QUICKTREE   :use FAST algorithm for the alignment guide tree
    -TYPE=       :PROTEIN or DNA sequences
    -NEGATIVE    :protein alignment with negative values in matrix
    -OUTFILE=    :sequence alignment file name
    -OUTPUT=     :GCG, GDE, PHYLIP, PIR or NEXUS
    -OUTORDER=   :INPUT or ALIGNED
    -CASE        :LOWER or UPPER (for GDE output only)
    -SEQNOS=     :OFF or ON (for Clustal output only)
    -SEQNO_RANGE=:OFF or ON (NEW: for all output formats) 
    -RANGE=m,n   :sequence range to write starting m to m+n. 
    
    ***Fast Pairwise Alignments:***
    -KTUPLE=n    :word size
    -TOPDIAGS=n  :number of best diags.
    -WINDOW=n    :window around best diags.
    -PAIRGAP=n   :gap penalty
    -SCORE       :PERCENT or ABSOLUTE
    
    ***Slow Pairwise Alignments:***
    -PWMATRIX=    :Protein weight matrix=BLOSUM, PAM, GONNET, ID or filename
    -PWDNAMATRIX= :DNA weight matrix=IUB, CLUSTALW or filename
    -PWGAPOPEN=f  :gap opening penalty        
    -PWGAPEXT=f   :gap opening penalty
    
    ***Multiple Alignments:***
    -NEWTREE=      :file for new guide tree
    -USETREE=      :file for old guide tree
    -MATRIX=       :Protein weight matrix=BLOSUM, PAM, GONNET, ID or filename
    -DNAMATRIX=    :DNA weight matrix=IUB, CLUSTALW or filename
    -GAPOPEN=f     :gap opening penalty        
    -GAPEXT=f      :gap extension penalty
    -ENDGAPS       :no end gap separation pen. 
    -GAPDIST=n     :gap separation pen. range
    -NOPGAP        :residue-specific gaps off  
    -NOHGAP        :hydrophilic gaps off
    -HGAPRESIDUES= :list hydrophilic res.    
    -MAXDIV=n      :% ident. for delay
    -TYPE=         :PROTEIN or DNA
    -TRANSWEIGHT=f :transitions weighting
    
    ***Profile Alignments:***
    -PROFILE      :Merge two alignments by profile alignment
    -NEWTREE1=    :file for new guide tree for profile1
    -NEWTREE2=    :file for new guide tree for profile2
    -USETREE1=    :file for old guide tree for profile1
    -USETREE2=    :file for old guide tree for profile2
    
    ***Sequence to Profile Alignments:***
    -SEQUENCES   :Sequentially add profile2 sequences to profile1 alignment
    -NEWTREE=    :file for new guide tree
    -USETREE=    :file for old guide tree
    
    ***Structure Alignments:***
    -NOSECSTR1     :do not use secondary structure-gap penalty mask for profile 1 
    -NOSECSTR2     :do not use secondary structure-gap penalty mask for profile 2
    -SECSTROUT=STRUCTURE or MASK or BOTH or NONE   :output in alignment file
    -HELIXGAP=n    :gap penalty for helix core residues 
    -STRANDGAP=n   :gap penalty for strand core residues
    -LOOPGAP=n     :gap penalty for loop regions
    -TERMINALGAP=n :gap penalty for structure termini
    -HELIXENDIN=n  :number of residues inside helix to be treated as terminal
    -HELIXENDOUT=n :number of residues outside helix to be treated as terminal
    -STRANDENDIN=n :number of residues inside strand to be treated as terminal
    -STRANDENDOUT=n:number of residues outside strand to be treated as terminal 
    
    ***Trees:***
    -OUTPUTTREE=nj OR phylip OR dist OR nexus
    -SEED=n        :seed number for bootstraps.
    -KIMURA        :use Kimura's correction.   
    -TOSSGAPS      :ignore positions with gaps.
    -BOOTLABELS=node OR branch :position of bootstrap values in tree display
    
    

    Batch Usage

    Sample batch script for the Oakley system:

    #PBS -N clustalw
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    
    cd $PBS_O_WORKDIR
    module load clustalw
    clustalw2 -INFILE=myfile.seqs -GAPOPEN=2 -GAPEXT=4

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    Cufflinks

    Cufflinks is a program that analyzes RNA -Seq samples. It assembles aligned RNA-Seq reads into a set of transcripts, then inspects the transcripts to estimate abundances and test for differential expression and regulation in the RNA-Seq reads.

    Availability and Restrictions

    Versions

    Cufflinks is available on the Owens Cluster. The versions currently available at OSC are:

    VERSION OWENS
    2.2.1 X*
    * Current Default Version

    You can use module spider cufflinksto view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Cufflinks is available without restriction to all OSC users.

    Usage

    Usage on Owens

    Set-up

    To configure your enviorment for use of Cufflinks, use command module load cufflinks. This will load the default version.

    Further Reading

    Fields of Science: 

    Darshan

    Darshan is a lightweight "scalable HPC I/O characterization tool".  It is intended to profile I/O by emitting log files to a consistent log location for systems administrators, and also provides scripts to create summary PDFs to characterize I/O in MPI-based programs.

    Availability and Restrictions

    Versions

    The following versions of Darshan are available on OSC clusters:

    version Owens
    3.1.2 X

    Access 

    Darshan is available to all OSC users without restriction.

    Usage on Owens

    Setup on Owens

    To configure the Owens cluster for Darshan use the following commands:

    module load darshan
    

    Darshan is only supported for the following compiler and MPI implementations:

    gnu/4.8.5  mvapich2/2.2
    gnu/4.8.5  mvapich2/2.2rc1
    gnu/4.8.5  openmpi/1.10
    intel/16.0.3  intelmpi/5.1.3
    intel/16.0.3  mvapich2/2.2
    intel/16.0.3  mvapich2/2.2rc1
    intel/16.0.3  openmpi/1.10
    

    Batch Usage on Owens

    Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for Owens, and Scheduling Policies and Limits for more info. 

    If you have an MPI-based program the syntax is as simple as

    # basic call to darshan 
    mpiexec.darshan [args] ./my_mpi_program
    # to show evidence that Darshan is working and to see internal timing
    mpiexec.darshan.timing [args] ./my_mpi_program
    
    An Example of Using Darshan with MPI-IO

    Below is an example batch script ( mpiio_with_darshan.qsub ) for understanding MPI-IO, see this resource for a detailed explanation: http://beige.ucs.indiana.edu/I590/node29.html.  The C program examples have each MPI task write to the same file at different offset locations sequentially.  A serial version (1 processor writes to 1 file) is included and timed for comparison.  Because the files generated here are large scratch files there is no need to retain them.

    1. Load the Darshan module
    2. Create a function as a build and run harness for each case
    3. Run the cases
    4. Generate PDF reports (using pdflatex and supporting files) from binary log files.
    5. Check job output and read PDF reports.
    #!/bin/bash
    #PBS -l nodes=1:ppn=28:pfsdir
    #PBS -j oe
    # one may need to perform 'shopt -s expand_aliases' inside the shell before calling this script to expand darshan aliases
    shopt -s expand_aliases
    ml r --quiet
    module load darshan
    ml
    function run_darshan() {
    COMPILER=mpicc
    MPIPROCS=$1
    PROGNAME=$2
    BLOKSIZE=$3
    LOCATION=$4
    cp ${DARSHAN_EXAMPLE}/${PROGNAME}.c $LOCATION
    cd $LOCATION
    $COMPILER -DDEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -o $PROGNAME ${PROGNAME}.c
    mpiexec_darshan() { eval mpiexec.darshan "$@"; }
    mpiexec_darshan -n $MPIPROCS ./$PROGNAME -f test -l $BLOKSIZE
    du -sh test # show the human-readable file size
    rm test # we are not keeping the large file around
    }
    # run the parallel version of mpi-io to write to a file but use the local tmp directory
    run_darshan $PBS_NP mkrandpfile 256 $TMPDIR
    # run the parallel version of mpi-io to write to a file but use the parallel file system scratch location
    run_darshan $PBS_NP mkrandpfile 256 $PFSDIR
    # for each darshan log generate a PDF report
    JOBID=$([[ $tmp =~ ([0-9]*)\..* ]]; echo "${BASH_REMATCH[1]}")
    for log in $(ls $DARSHAN_LOG/$(date "+%Y/%-m/%-d")/${USER:0:8}_*_id${JOBID}*.darshan); do
    darshan-job-summary.pl $log
    done
    cp *.pdf $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the mpiio_with_darshan.qsub   file with the following command:

    qsub mpiio_with_darshan.qsub
    

    Further Reading

    http://www.mcs.anl.gov/research/projects/darshan/

    Service: 
    Technologies: 
    Fields of Science: 

    EMBOSS

    EMBOSS is "The European Molecular Biology Open Software Suite". EMBOSS is a free Open Source software analysis package specially developed for the needs of the molecular biology (e.g. EMBnet) user community. The software automatically copes with data in a variety of formats and even allows transparent retrieval of sequence data from the web. Also, as extensive libraries are provided with the package, it is a platform to allow other scientists to develop and release software in true open source spirit. EMBOSS also integrates a range of currently available packages and tools for sequence analysis into a seamless whole.

    Within EMBOSS you will find around hundreds of programs (applications) covering areas such as:

    • Sequence alignment,
    • Rapid database searching with sequence patterns,
    • Protein motif identification, including domain analysis,
    • Nucleotide sequence pattern analysis---for example to identify CpG islands or repeats,
    • Codon usage analysis for small genomes,
    • Rapid identification of sequence patterns in large scale sequence sets,
    • Presentation tools for publication

    Availability & Restrictions

    The EMBOSS software package is available without restriction for all academic OSC users.

    The following versions of EMBOSS are available on OSC systems:

    Version Oakley
    6.4.0 X

    Usage on Oakley

    Set-up

    For the Oakley system, run the following command:

    module load emboss
    

    Using EMBOSS

    The EMBOSS programs are typically used to perform one or more tasks on a large number of sequences.  Once the emboss module is loaded, all emboss commands are available for your use.

    Batch Usage

    Coming soon.

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    Elmer

    Elmer is an open source multiphysical simulation software developed by CSC - IT (Finland).

    From their Github README:

    Elmer is a finite element software for numerical solution of partial
    differential equations. Elmer is capable of handling any number of equations
    and is therefore ideally suited for the simulation of multiphysical problems.
    It includes models, for example, of structural mechanics, fluid dynamics, heat
    transfer and electromagnetics. Users can also write their own equations that
    can be dynamically linked with the main program.
    Elmer consists of several parts. The most important ones are ElmerSolver, the
    finite element solver, ElmerGUI, the graphical user interface, and ElmerGrid,
    the mesh creation and manipulation tool. Also a visualization tool, ElmerPost,
    is included in the package but it is no longer developed.  
    Elmer software is licensed under GPL except for the ElmerSolver library which
    is licensed under LGPL license. 
    Elmer is mainly developed at CSC - IT Center for Science, Finland. However,
    there have been numerous contributions from other organizations and developers
    as well, and the project is open for new contributions.
    

    Availability and Restrictions

    Versions

    The following versions of Elmer are available on OSC clusters:

    version oakley Ruby
    8.1 X X

     

    The current versions of Elmer are compiled with GNU 4.8.4 (requirement > 4.8) and BLAS libraries from MKL/11.2.3.

    Feel free to contact OSC Help if you need other versions for your work.

    Access 

    Elmer is available to all OSC users without restriction.

    Usage

    Usage on Oakley or Ruby

    Setup on Oakley or Ruby

    To configure the Oakley or Ruby cluster for the use of Elmer 8.1, use the following commands:

    module load gnu/4.8.4
    module load mkl/11.2.3
    module load elmer
    

    Batch Usage on Oakley or Ruby

    Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations for OakleyQueues and Reservations for Ruby, and Scheduling Policies and Limits for more info.

    Converting Mesh with ElmerGrid and Running ElmerSolver

    Below is an example batch script ( job.txt ) for using Elmer:

    #PBS -N Elmer
    #PBS -l nodes=1:ppn=1 
    #PBS -l walltime=30:00
    #PBS -j oe 
    #PBS -S /bin/bash 
    # Load the modules for Elmer
    module load gnu/4.8.4
    module load mkl/11.2.3
    module load elmer
    # Move to the problem directory
    cd $PBS_O_WORKDIR
    # Copy files to $TMPDIR and move there to execute the program
    cp -r * $TMPDIR
    cd $TMPDIR
    # Convert the mesh - see Elmer webpage for more examples of 'ElmerGrid'
    ElmerGrid 14 2 beam.msh -autoclean
    # Run the solver 
    ElmerSolver beam.sif
    # Finally, copy files back to your home directory 
    cp -r * $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt   file with the following command:

    qsub job.txt
    
    

    Further Reading

    https://www.csc.fi/web/elmer/

    https://github.com/ElmerCSC/elmerfem

    Service: 
    Fields of Science: 

    FASTX-Toolkit

    The FASTX-Toolkit is a collection of command line tools for Short-Reads FASTA/FASTQ files preprocessing.

    Availability and Restrictions

    The following versions of FASTX-Toolkit are available on OSC clusters:

    Version Oakley Owens
    0.0.14 X* X*
    *: Current default version

    You can use  module spider fastx to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    FASTX-Toolkit is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of FASTX-Toolkit, run the following command:   module load fastx. The default version will be loaded. To select a particular FASTX-Toolkit version, use   module load fastx/version. For example, use  module load fastx/0.0.14 to load FASTX-Toolkit 0.0.14.

    Usage on Owens

    Set-up

    To configure your environment for use of FASTX-Toolkit, run the following command:   module load fastx. The default version will be loaded. To select a particular FASTX-Toolkit version, use   module load fastx/version. For example, use  module load fastx/0.0.14 to load FASTX-Toolkit 0.0.14.

    Further Reading

    Service: 
    Fields of Science: 

    FFTW

    FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data. It is portable and performs well on a wide variety of platforms.

    Availability and Restrictions

    FFTW is available on Oakley, Ruby, and Owens Clusters. The versions currently available at OSC are:

    Version OAKLEY RUBY Owens notes
    2.1 X      
    2.1.5   X(GI)    
    3.1.2        
    3.3 X     Default version on Oakley prior to 09/15/2015
    3.3.1        
    3.3.2        
    3.3.3 X(I)      
    3.3.4 X X X(GI)  
    3.3.5 X* X* X*  
    • X(G): gnu versions only,
    • X(I): intel version only
    • X(GI): available with both gnu and intel
    • X: available with gnu, intel, and pgi
    * : Current Default Version (note: FFTW2 and FFTW3 are tracked separately in the module system)

    You can use module spider fftw2  or module spider fftw3   to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    FFTW is available without restriction to all OSC Users

    Usage on Oakley

    Set-up

    Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module load name  . For example, use  module load fftw3/3.3.3  to load FFTW3 version 3.3.3. You can use module spider fftw  to view available modules.

    Building with FFTW

    The following environment variables are setup when the FFTW library is loaded:

    Variable Use
    $FFTW3_CFLAGS Use during your compilation step for C programs.
    $FFTW3_FFLAGS Use during your compilation step for Fortran programs.
    $FFTW3_LIBS Use during your link step for the sequential version of the library.
    $FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
    $FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.

    below is a set of example commands used to build a file called my-fftw.c .

    module load fftw3
    icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
    ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS
    

    Usage on Ruby

    Set-up

    Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using.  A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module load name  . For example, use  module load fftw3/3.3.4  to load FFTW3 version 3.3.4. You can use module spider fftw  to view available modules.

    Building with FFTW

    The following environment variables are setup when the FFTW library is loaded:

    Variable Use
    $FFTW3_CFLAGS Use during your compilation step for C programs.
    $FFTW3_FFLAGS Use during your compilation step for Fortran programs.
    $FFTW3_LIBS Use during your link step for the sequential version of the library.
    $FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
    $FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.

    below is a set of example commands used to build a file called my-fftw.c .

    module load fftw3
    icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
    ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS
    

    Usage on Owens

    Set-up

    Initalizing the system for use of the FFTW library is dependent on the system you are using and the compiler you are using. A successful build of your program will depend on an understanding of what module fits your circumstances. To load a particular version, use  module load name  . For example, use  module load fftw3/3.3.4  to load FFTW3 version 3.3.4. You can use module spider fftw  to view available modules.

    Building with FFTW

    The following environment variables are setup when the FFTW library is loaded:

    Variable Use
    $FFTW3_CFLAGS Use during your compilation step for C programs.
    $FFTW3_FFLAGS Use during your compilation step for Fortran programs.
    $FFTW3_LIBS Use during your link step for the sequential version of the library.
    $FFTW3_LIBS_OMP Use during your link step for the OpenMP version of the library.
    $FFTW3_LIBS_MPI Use during your link step for the MPI version of the library.
    $FFTW3_LIBS_THREADS Use during your link step for the "threads" version of the library.

    below is a set of example commands used to build a file called my-fftw.c .

    module load fftw3
    icc $FFTW3_CFLAGS my-fftw.c -o my-fftw $FFTW3_LIBS 
    ifort $FFTW3_FFLAGS more-fftw.f -o more-fftw $FFTW3_LIBS
    

    Further Reading

    See Also

    Service: 

    FSL

    FSL is a library of tools for analyzing FMRI, MRI and DTI brain imaging data.

    Availability & Restrictions

    Versions

    The following versions of FSL are available on OSC clusters:

    VERSION OAKLEY RUBY OWENS
    5.0.10    

    X*

    * Curent default version

    Access

    FSL is available to all academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction.

    Usage

    Usage on Owens

    Set-up

    Configure your environment for use of FSL with module load fsl. This will load the default version.

    Using FSL GUI

    Access the FSL GUI with command

    fsl

    This will bring up a menu of all FSL tools. For information on individual FSL tools see FSL Overview page.

     

    Further Reading 

     

    Fields of Science: 

    FreeSurfer

    FreeSurfer is a software package used to anaylze nueroimaging data.

    Availability & Restrictions

    Versions

    The following versions of FreeSurfer are available on OSC clusters:

    VERSION OAKLEY RUBY OWENS
    6.0.0    

    X*

    * Curent default version

    Access

    FreeSurfer is available to all academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction. 

    Usage

    Usage on Owens

    Set-up

    Load the FreeSurfer module with  module load freesurfer . This will load the default version. Then, to continue configuring your environment, you must source the setup script for Freesurfer. Do this with the following command that corresponds to the Linux shell you are using. If using bash, use

    source $FREESURFER_HOME/SetUpFreeSurfer.sh
    

    If using tcsh, use

    source $FREESURFER_HOME/SetUpFreeSurfer.csh
    

    To finish configuring FreeSurfer, set the the FreeSurfer environment variable SUBJECTS_DIR  to the directory of your subject data. The SUBJECTS_DIR variable defaults to the FREESURFER_HOME/subjects directory, so if this is your intended directory to use the enviornment set-up is complete.

    To alter the SUBJECTS_DIR variable, however, again use the following command that corresponds to the Linux shell you are using. For bash, 

    export SUBJECTS_DIR=<path to subject data>
    

    For tcsh

    setenv SUBJECTS_DIR=<path to subject data>
    

    Note that you can set the SUBJECT_DIR variable before or after sourcing the setup script.

     

    Further Reading 

    Service: 
    Fields of Science: 

    FreeType

    FreeType is a freely available software library to render fonts. It is written in C, designed to be small, efficient, highly customizable, and portable while capable of producing high-quality output (glyph images) of most vector and bitmap font formats.

    Availability and Restrictions

    Versions

    The following versions of FreeType are available on OSC clusters:

    VERSION OAKLEY
    2.5.5 X*
    *: Default Version

    You can use  module spider freetype  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    FreeType is available to all OSC users without restriction.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To configure your environment for use of FreeType on Oakley, run the following command:  module load freetype . The default version will be loaded. To select a particular FreeType version, use  module load freetype/version . For example, use module load freetype/2.5.5  to load FreeType 2.5.5 on Oakley.

    Further Reading

    Service: 
    Fields of Science: 

    GAMESS

    Introduction

    The General Atomic and Molecular Electronic Structure System (GAMESS) is a flexible ab initio electronic structure program. Its latest version can perform general valence bond, multiconfiguration self-consistent field, Möller-Plesset, coupled-cluster, and configuration interaction calculations. Geometry optimizations, vibrational frequencies, thermodynamic properties, and solution modeling are available. It performs well on open shell and excited state systems and can model relativistic effects. The GAMESS Home Page has additional information.

    Availability

    GAMESS is available on the Oakley and Owens Clusters.

    Version

     

    VERSION

    OAKLEY owens

    1 MAY 2013 (R1)

    X*

     
    18 AUG 2016 (R1)   X*

    * - Default Version

    Usage

    Users are not required to complete a license agreement.

    Location

    The root of the GAMESS directory tree is /usr/local/gamess/

    Execution

    GAMESS usage is controlled via modules. Load one of the GAMESS modulefiles at the command line, in your shell initialization script, or in your batch scripts, for example:

    module load gamess  

    Examples

    Further Reading

    General documentation is available from the GAMESS Home page and in the local machine directories.

    Service: 

    GATK

    GATK is a software package for analysis of high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance.

    Availability and Restrictions

    The following versions of GATK are available on OSC clusters:

    Version Oakley Owens
    3.5 X* X*
    *: Current default version

    You can use  module spider gatk to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    GATK is available to academic OSC users with proper validation. In order to obtain validation, please contact OSC Help for further instruction. 

    Usage on Oakley

    Set-up

    To configure your environment for use of GATK, run the following command:   module load gatk. The default version will be loaded. To select a particular GATK version, use   module load gatk/version. For example, use  module load gatk/3.5to load GATK 3.5.

    Usage

    This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable.

    From module load gatk , a new environment variable, GATK, will be set.

    Thus, users can use the software by running the following command: java -jar $GATK {other options}

    Usage on Owens

    Set-up

    To configure your environment for use of GATK, run the following command:   module load gatk. The default version will be loaded. To select a particular GATK version, use   module load gatk/version. For example, use module load gatk/3.5 to load GATK 3.5.

    Usage

    This software is a Java executable .jar file; thus, it is not possible to add to the PATH environment variable.

    From module load gatk, a new environment variable, GATK, will be set.

    Thus, users can use the software by running the following command: java -jar $GATK {other options}

    Further Reading

    Service: 
    Fields of Science: 

    GLPK

    GLPK (GNU Linear Programming Kit) is a set of open source LP (linear programming) and MIP (mixed integer problem) routines written in ANSI C, which can be called from within C programs. 

    Availability & Restrictions

    GLPK is available to all OSC users without restriction. 

    The following versions are available on OSC systems:

    version oakley
    4.48 X

    Usage

    Setup

    To set up your environment for using GLPK on Oakley, run the following command:

    module load glpk

    Compiling and Linking

    To compile your C code using GLPK API routines, use the environment variable $GLPK_CFLAGS provided by the module:

    gcc $GLPK_CFLAGS -c my_prog.c

    To link your code, use the variable $GLPK_LIBS:

    gcc my_prog.o $GLPK_LIBS -o my_prog

    glpsol

    Additionally, the GLPK module contains a stand-alone LP/MIP solver, which can be used to process files written in the GNU MathProg modeling language.  The solver can be invoked using the following command syntax:

    glpsol [options] [filename]

    For a complete list of options, use the following command:

    glpsol --help

    Further Reading

    GLPK Homepage

    Service: 

    GMAP

    GMAP: a genomic mapping and alignment program for mRNA and EST sequences.

    Availability and Restrictions

    The following versions of GMAP are available on OSC clusters:

    Version Oakley Owens
    2016-05-25 X*  
    2016-06-09   X*
    *: Current default version

    You can use  module spider gmap   to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    GMAP is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of GMAP, run the following command:   module load gmap . The default version will be loaded. To select a particular GMAP version, use   module load gmap/version . For example, use  module load gmap/2016-05-25  to load GMAP 2016-05-25.

    Usage on Owens

    Set-up

    To configure your environment for use of GMAP, run the following command:   module load gmap . The default version will be loaded. To select a particular GMAP version, use   module load gmap/version . For example, use module load gmap/2016-06-09  to load GMAP 2016-06-09.

    Further Reading

    Service: 
    Fields of Science: 

    GNU Compilers

    Fortran, C and C++ compilers produced by the GNU Project. 

    Availability and Restrictions

    GNU Compiler is available on Oakley, Ruby, and Owens Clusters. The versions currently available at OSC are:

    Version Oakley Ruby Owens notes
    4.4.7 X# X#   Default version on Oakley and Ruby prior to 09/15/2015
    4.8.4 X X    
    4.8.5 X* X* X*# **See note below.
    4.9.1 X X    
    5.1.0 X      
    5.2.0 X X    
    6.1.0     X  
    6.3.0 X X X Libraries have been built for this version
    * : Current Default Version
    # : System version
    ** There is always some version of the GNU compilers in the environment. If you want a specific version you should load the appropriate module. If you don't have a gnu module loaded you will get either the system version or some other version, depending on what modules you do have loaded.

    You can use module spider gnu  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    To find out what version of gcc you are using, type gcc --version.

    Access

    The GNU compilers are available to all OSC users without restriction.

    Usage

    Usage on Oakley

    Set-up

    To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions):  module load gnu . The default version will be loaded. To select a particular GNU version, use module load gnu/version . For example, use module load gnu/4.8.4  to load GNU 4.8.4.

    How to Compile

    Once the module is loaded, follow the guides below for compile commands:

    Language non-mpi mpi
    Fortran 90 or 95 gfortran mpif90
    Fortran 77 g77 mpif77
    c gcc mpicc
    c++ g++ mpicxx

    Building Options

    The GNU compilers recognize the following command line options :

    Compiler Option Purpose
    -fopenmp Enables compiler recognition of OpenMPdirectives (except g77 and mpi77)
    -o FILENAME

    Specifies the name of the object file

    -O0 or no -O  option Disable optimization
    -O1 or -O Ligh optimization
    -O2 Heavy optimization
    -O3 Most expensive optimization (Recommended)

    There are numerous flags that can be used. For more information run man <compiler binary name> .

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c  and the output file named hello_results
    Below is the example batch script ( job.txt ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=12
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    cp hello.c $TMPDIR
    cd $TMPDIR
    gcc -O3 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script ( job.txt ) for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    mpicc -O3 hello.c -o hello
    pbsdcp hello $TMPDIR
    cd $TMPDIR
    mpiexec hello > hello_results
    pbsdcp -g hello_results $PBS_O_WORKDIR
    

    Usage on Ruby

    Set-up

    To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions):   module load gnu .  The default version will be loaded. To select a particular GNU version, use  module load gnu/version  . For example, use module load gnu/4.9.1  to load GNU 4.9.1.

    How to Compile

    Once the module is loaded, follow the guides below for compile commands:

    Language non-mpi mpi
    Fortran 90 or 95 gfortran mpif90
    Fortran 77 g77 mpif77
    c gcc mpicc
    c++ g++ mpicxx

    Building Options

    The GNU compilers recognize the following command line options :

    Compiler Option Purpose
    -fopenmp Enables compiler recognition of OpenMPdirectives (except g77 and mpi77)
    -o FILENAME

    Specifies the name of the object file

    -O0 or no -O  option Disable optimization
    -O1 or -O Ligh optimization
    -O2 Heavy optimization
    -O3 Most expensive optimization (Recommended)

    There are numerous flags that can be used. For more information run man <compiler binary name> .

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00
    
    which gives you 20 cores (  -l nodes=1:ppn=20 )  with 1 hour (  -l walltime=1:00:00 ).  You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results
    Below is the example batch script (  job.txt  ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=20
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    cp hello.c $TMPDIR
    cd $TMPDIR
    gcc -O3 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the  job.txt  file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script (  job.txt  ) for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=20
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    mpicc -O3 hello.c -o hello
    cp hello $TMPDIR
    cd $TMPDIR
    mpiexec ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    Usage on Owens

    Set-up

    To configure your environment for use of the GNU compilers, run the following command (you may have to unload your selected compiler - if an error message appears, it will provide instructions):   module load gnu .  The default version will be loaded. To select a particular GNU version, use  module load gnu/version  . For example, use module load gnu/4.8.5  to load GNU 4.8.5.

    How to Compile

    Once the module is loaded, follow the guides below for compile commands:

    Language non-mpi mpi
    Fortran 90 or 95 gfortran mpif90
    Fortran 77 g77 mpif77
    c gcc mpicc
    c++ g++ mpicxx

    Building Options

    The GNU compilers recognize the following command line options :

    Compiler Option Purpose
    -fopenmp Enables compiler recognition of OpenMPdirectives (except g77 and mpi77)
    -o FILENAME

    Specifies the name of the object file

    -O0 or no -O  option Disable optimization
    -O1 or -O Ligh optimization
    -O2 Heavy optimization
    -O3 Most expensive optimization (Recommended)

    There are numerous flags that can be used. For more information run man <compiler binary name> .

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores (  -l nodes=1:ppn=28 )  with 1 hour (  -l walltime=1:00:00 ).  You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results
    Below is the example batch script (  job.txt  ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=28
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    cp hello.c $TMPDIR
    cd $TMPDIR
    gcc -O3 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the  job.txt  file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script (  job.txt  ) for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=28
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load gnu
    mpicc -O3 hello.c -o hello
    cp hello $TMPDIR
    cd $TMPDIR
    mpiexec ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    Further Reading

    See Also

    Service: 
    Technologies: 
    Fields of Science: 

    GPU-BLAST

    GPU-BLAST is an accelerated version of the popular NCBI-BLAST (www.ncbi.nlm.nih.gov) that uses general-purpose graphics processing unit (GPU). In comparison to the sequential NCBI-BLAST, GPU-BLAST is nearly four times faster, while producing identical results.

    Availability & Restrictions

    GPU-BLAST is available without restriction to all OSC users.

    The following versions of GPU-BLAST are available on OSC systems:

    VERSION NCBI-BLAST VERSION OAKLEY RUBY
    1.0 2.2.24+ X  
    1.1 2.2.25+ X  
    1.1 2.2.26+  X*  
    1.1 2.2.28+   X*

     

    * Current default version

    Note that GPU-BLAST is not available on the Owens system. GPU-BLAST requires the CUDA nvcc compiler, but the GPU-BLAST software is only compatible with CUDA versions up to 7.5. CUDA 8, installed on the Owens cluster, is incompatible with GPU-BLAST.

    Usage

    Usage on Oakley

    Set-up

    To set up the environment for gpu-blast on Ruby cluster, do the following:

    module load cuda/4.1.28
    module load gpu-blast
    

    Batch Usage

    Provided is a sample batch script:

    #PBS -l nodes=1:ppn=1:gpus=1
    #PBS -l walltime=10:00
    #PBS -N GPU-Blast
    #PBS -S /bin/bash
    #PBS -j oe
    module load gpu-blast
    module load cuda/4.1.28
    set -x
    cd $PBS_O_WORKDIR
    mkdir $PBS_JOBID
    cp 100.fasta $TMPDIR
    cd $TMPDIR
    /usr/bin/time blastn -db nt -query 100.fasta -out test.out
    cp * $PBS_O_WORKDIR/$PBS_JOBID
    
     

    Usage on Ruby

    Set-up

    To set up the environment for gpu-blast on Ruby cluster, do the following:

    module load cuda/7.5.18
    module load gpu-blast
    

    Batch Usage

    Provided is a sample batch script:

    #PBS -l nodes=1:ppn=1:gpus=1
    #PBS -l walltime=10:00
    #PBS -N GPU-Blast
    #PBS -S /bin/bash
    #PBS -j oe
    module load gpu-blast
    module load cuda/7.5.18
    set -x
    cd $PBS_O_WORKDIR
    mkdir $PBS_JOBID
    cp 100.fasta $TMPDIR
    cd $TMPDIR
    /usr/bin/time blastn -db nt -query 100.fasta -out test.out
    cp * $PBS_O_WORKDIR/$PBS_JOBID
    
     

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    GROMACS

    GROMACS is a versatile package of molecular dynamics simulation programs. It is primarily designed for biochemical molecules, but it has also been used on non-biological systems.  GROMACS generally scales well on OSC platforms. Starting with version 4.6 GROMACS includes GPU acceleration.

    Availability and Restrictions

    GROMACS is available on Oakley and Owens Clusters. The versions currently available at OSC are the following (S means serial single node executables, P means parallel multinode, and C means CUDA, i.e., GPU enabled):

    Version Oakley Owens notes
    4.5.5 SP   Default version on Oakley prior to 09/15/2015
    4.6.3 SPC   Default version on Oakley prior to 02/21/2017
    5.1 SPC*    
    5.1.2   SPC*  
    2016.4   SPC  
    *: Current default version

    You can use module spider gromacs  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    GROMACS is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To load the module for the default version of GROMACS, which initializes your environment for the GROMACS application, use  module load gromacs . To select a particular software version, use   module load gromacs/version . For example, use  module load gromacs/5.1  to load GROMACS version 5.1; and use  module help gromacs/5.1  to view details, such as, compiler prerequisites, additional modules required for specific executables, the suffixes of executables, etc.

    Using GROMACS

    The syntax of the GROMACS executable(s) changed in version 5. To execute a serial GROMACS versions 4 program interactively, simply run it on the command line, e.g.:
    pdb2gmx
    

    To execute a serial GROMACS versions 5 program interactively, simply run it on the command line, e.g.:

    gmx pdb2gmx
    

    Note that some serial executables are enabled for single node parallelism via OpenMP.  Parallel multinode GROMACS versions 4 programs should be run in a batch environment with mpiexec, e.g.:

    mpiexec mdrun_mpi_d
    

    Parallel multinode GROMACS versions 5 programs should be run in a batch environment with mpiexec, e.g.:

    mpiexec gmx_mpi_d mdrun
    

    Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision ('_gpu' denotes a GPU executable built with CUDA).  See the module help output for specific versions for more details on executable naming conventions.

    Batch Usage

    When you log into Oakley you are actually connected to a login node. To  access the compute nodes, you must submit a job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Oakley, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Parallel Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and input files are available here:

    ~srb/workshops/compchem/gromacs/
    

    This simple batch script demonstrates some important points:

    # GROMACS Tutorial for Solvation Study of Spider Toxin Peptide
    # see fwspider_tutor.pdf
    #PBS -N fwsinvacuo.oakley
    #PBS -l nodes=2:ppn=12
    module unload intel
    module load intel/12.1.4.319
    module load mvapich2/1.7
    module load gromacs/4.6.3
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cd $PBS_O_WORKDIR
    pbsdcp -p 1OMB.pdb em.mdp $TMPDIR
    # Use TMPDIR for best performance.
    cd $TMPDIR
    pdb2gmx -ignh -ff gromos43a1 -f 1OMB.pdb -o fws.gro -p fws.top -water none
    editconf -f fws.gro -d 0.7
    editconf -f out.gro -o fws_ctr.gro -center 2.0715 1.6745 1.914
    grompp -f em.mdp -c fws_ctr.gro -p fws.top -o fws_em.tpr
    mpiexec mdrun_mpi -s fws_em.tpr -o fws_em.trr -c fws_ctr.gro -g em.log -e em.edr
    cat em.log
    cp -p * $PBS_O_WORKDIR/
    

    Usage on Owens

    Set-up

    To load the module for the default version of GROMACS, which initializes your environment for the GROMACS application, use module load gromacs . To select a particular software version, use   module load gromacs/version . For example, use  module load gromacs/5.1.2  to load GROMACS version 5.1.2; and use  module help gromacs/5.1.2  to view details, such as, compiler prerequisites, additional modules required for specific executables, the suffixes of executables, etc.

    Using GROMACS

    To execute a serial GROMACS versions 5 program interactively, simply run it on the command line, e.g.:

    gmx pdb2gmx
    

    Parallel multinode GROMACS versions 5 programs should be run in a batch environment with mpiexec, e.g.:

    mpiexec gmx_mpi_d mdrun
    

    Note that '_mpi' indicates a parallel executable and '_d' indicates a program built with double precision ('_gpu' denotes a GPU executable built with CUDA).  See the module help output for specific versions for more details on executable naming conventions.

    Batch Usage

    When you log into Owens you are actually connected to a login node. To  access the compute nodes, you must submit a job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Owens, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores ( -l nodes=1:ppn=28 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Parallel Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and input files are available here:

    ~srb/workshops/compchem/gromacs/
    

    This simple batch script demonstrates some important points:

    # GROMACS Tutorial for Solvation Study of Spider Toxin Peptide
    # see fwspider_tutor.pdf
    #PBS -N fwsinvacuo.owens
    #PBS -l nodes=2:ppn=28
    module load gromacs
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cd $PBS_O_WORKDIR
    pbsdcp -p 1OMB.pdb em.mdp $TMPDIR
    # Use TMPDIR for best performance.
    cd $TMPDIR
    pdb2gmx -ignh -ff gromos43a1 -f 1OMB.pdb -o fws.gro -p fws.top -water none
    editconf -f fws.gro -d 0.7
    editconf -f out.gro -o fws_ctr.gro -center 2.0715 1.6745 1.914
    grompp -f em.mdp -c fws_ctr.gro -p fws.top -o fws_em.tpr
    mpiexec gmx_mpi mdrun -s fws_em.tpr -o fws_em.trr -c fws_ctr.gro -g em.log -e em.edr
    cat em.log
    cp -p * $PBS_O_WORKDIR/

    Further Reading

    Service: 

    Gaussian

    Gaussian is the most popular general purpose electronic structure program. Its latest version, g09, can perform density functional theory, Hartree-Fock, Möller-Plesset, coupled-cluster, and configuration interaction calculations among others. Geometry optimizations, vibrational frequencies, magnetic properties, and solution modeling are available. It performs well as black-box software on closed-shell ground state systems. 

    Availability and Restrictions

    Gaussian is available on Ruby, Oakley, and Owen Clusters. The versions currently available at OSC are:

    Version Oakley Ruby OWENS notes
    g03d01        
    g03e01        
    g09a01        
    g09b01 X      
    g09c01

    X

        Default version on Oakley prior to 09/15/2015
    g09d01 X X    
    g09e01 X* X* X*  

    g16a03

    g16a03-GPU

     

    X

    X

    X

     

     
    *: Current default version

    You can use module spider gaussian  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    Use of Gaussian for academic purposes requires validation. In order to obtain validation, please contact OSC Help for further instruction.

    Access for Commerical Users

    Contact OSC Help for getting access to Gaussian if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To load the default version of Gaussian module/initalize your environement for use of Gaussian, use  module load gaussian . To select a particular software version, use   module load gaussian/version . For example, use  module load gaussian/g09b01  to load Gaussian version g09b01 on Oakley. 

    Using Gaussian

    To execute Gaussian, simply run the Gaussian binary with the input file on the command line:

    g09 < input.com
    

    When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen via 'qpeek jobid' when the job is running in a batch queue.  Alternatively, Gaussian can be invoked without file redirection:

    g09 input.com
    

    in which case the output file will be named 'input.log'; in this form the output cannot be seen via 'qpeek jobid' when the job is running in a batch queue.

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Oakley, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

    /users/appl/srb/workshops/compchem/gaussian/
    

    This simple batch script demonstrates the important points:

    #PBS -N GaussianJob
    #PBS -l nodes=1:ppn=1
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cd $PBS_O_WORKDIR
    cp input.com $TMPDIR
    # Use TMPDIR for best performance.
    cd $TMPDIR
    module load gaussian
    g16 < input.com
    cp -p input.log *.chk $PBS_O_WORKDIR
    
    Note: OSC does not have a functional distributed parallel version (LINDA) of Gaussian. Parallelism of Gaussian at OSC is only via shared memory. Consequently, do not request more than one node for Gaussian jobs on OSC's clusters.

    Usage on Ruby

    Set-up on Ruby

    To load the default version of Gaussian module/initalize your environement for use of Gaussian, use  module load gaussian . To select a particular software version, use   module load gaussian/version . For example, use  module load gaussian/g09d01  to load Gaussian version g09d01 on Ruby. 

    Using Gaussian

    To execute Gaussian, simply run the Gaussian binary with the input file on the command line:

    g09 < input.com
    

    When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen via 'qpeek jobid' when the job is running in a batch queue.  Alternatively, Gaussian can be invoked without file redirection:

    g09 input.com
    

    in which case the output file will be named 'input.log'; in this form the output cannot be seen via 'qpeek jobid' when the job is running in a batch queue.

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Ruby, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00
    
    which gives you 20 cores ( -l nodes=1:ppn=20 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

    /users/appl/srb/workshops/compchem/gaussian/
    

    This simple batch script demonstrates the important points:

    #PBS -N GaussianJob
    #PBS -l nodes=1:ppn=20
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cd $PBS_O_WORKDIR
    cp input.com $TMPDIR
    # Use TMPDIR for best performance.
    cd $TMPDIR
    module load gaussian
    g09 < input.com
    cp -p input.log *.chk $PBS_O_WORKDIR
    
    Note: OSC does not have a functional distributed parallel version (LINDA) of Gaussian. Parallelism of Gaussian at OSC is only via shared memory. Consequently, do not request more than one node for Gaussian jobs on OSC's clusters.

    Running Gaussian jobs with GPU

    Gaussian jobs can utilize K40 GPUs that are avaialabe on Ruby.  GPUs are not helpful for small jobs but are effective for larger molecules when doing DFT energies, gradients and frequencies (for both ground and excited states). They are also not used effectively by post-SCF calculations such as MP2 or CCSD. For more information, please visit   http://gaussian.com/relnotes/?tabid=2 

    A sample batch script for GPU run will look like as follows:

    #PBS -S /bin/tcsh
    #PBS -N methane 
    #PBS -o methane.log
    #PBS -l nodes=1:ppn=20:gpus=1:default
    set echo
    cd $TMPDIR
    set INPUT=methane.com
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cp $PBS_O_WORKDIR/$INPUT .
    module load gaussian/g16a03
    g16 < ./$INPUT
    ls -al
    cp *.chk $PBS_O_WORKDIR
    cp *.log $PBS_O_WORKDIR
    

     

    A sample input file for GPU run will look like as follows:

    %nproc=20
    %mem=8gb
    %CPU=0-19
    %GPUCPU=0=0
    %chk=methane.chk
    #b3lyp/6-31G(d) opt
    methane B3LYP/6-31G(d) opt freq
    0,1
    C        0.000000        0.000000        0.000000
    H        0.000000        0.000000        1.089000
    H        1.026719        0.000000       -0.363000
    H       -0.513360       -0.889165       -0.363000
    H       -0.513360        0.889165       -0.363000
    

    The above example will utilize CPUs indexed from 0 to 19th, but 0th CPU is associated with 0th GPU.

     

    Usage on Owens

    Set-up on Owens

    To load the default version of Gaussian module/initalize your environement for use of Gaussian, use   module load gaussian  . To select a particular software version, use    module load gaussian/version  . For example, use   module load gaussian/g09d01   to load Gaussian version g09d01 on Owens. 

    Using Gaussian

    To execute Gaussian, simply run the Gaussian binary with the input file on the command line:

    g09 < input.com
    

    When the input file is redirected as above ( < ), the output will be standard output; in this form the output can be seen via 'qpeek jobid' when the job is running in a batch queue.  Alternatively, Gaussian can be invoked without file redirection:

    g09 input.com
    

    in which case the output file will be named 'input.log'; in this form the output cannot be seen via 'qpeek jobid' when the job is running in a batch queue.

    Batch Usage on Owens

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Owens, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores (  -l nodes=1:ppn=28  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and Gaussian input files are available here:

    /users/appl/srb/workshops/compchem/gaussian/
    

    This simple batch script demonstrates the important points:

    #PBS -N GaussianJob
    #PBS -l nodes=1:ppn=28
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cd $PBS_O_WORKDIR
    cp input.com $TMPDIR
    # Use TMPDIR for best performance.
    cd $TMPDIR
    module load gaussian
    g09 < input.com
    cp -p input.log *.chk $PBS_O_WORKDIR
    
    Note: OSC does not have a functional distributed parallel version (LINDA) of Gaussian. Parallelism of Gaussian at OSC is only via shared memory. Consequently, do not request more than one node for Gaussian jobs on OSC's clusters.

    Further Reading

     

    Service: 

    Gnuplot

    Introduction

    Gnuplot is a portable command-line driven data and function plotting utility.  It was originally intended to allow scientists and students to visualize mathematical functions and data.  

    Gnuplot supports many types of plots in two or three dimensions.  It can draw using points, lines, boxes, contours, vector fields surfaces and various associated text.  It also supports various specialized plot types.  

    Gnuplot supports many different types of output:  interactive screen display (with mouse and hotkey functionality), pen plotters (like hpgl), printers (including postscript and many color devices), and file formats as vectorial pseudo-devices like LaTeX, metafont, pdf, svg, or bitmap png.  

    Version

     

    Version Oakley Owens Notes
    4.2.4      
    4.6.4 X    
    4.6 patchlevel 2   System Install No Module Needed

    Usage on Oakley

    To start a Gnuplot session, load the module and launch using the following commands:

    module load gnuplot
    
    gnuplot
    
    

    To access the Gnuplot help menu, type ? into the Gnuplot command line.  

    Usage on Owens

    Since Gnuplot is a System Install, no module is needed to use the application.

    Further Reading

    For more information, visit the Gnuplot Homepage.  

    Service: 

    HDF5

    HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

    Availability and Restrictions

    HDF5 is available on Oakley, Ruby, and Owens Clusters. The versions currently available at OSC are:

    Version Oakley Ruby Owens notes
    1.6.4        
    1.6.5        
    1.8.7        
    1.8.8 X      
    1.8.12   X(GI)   Default version on Ruby prior to 09/14/2015
    1.8.13 X(GI) X(GI)

     

    Default version on Oakley prior to 09/14/2015
    1.8.15 X X(PI)    
    1.8.17 X* X* X*  
    • X(P): pgi only
    • X(PI): available with pgi and intel
    • X(GI): available with gnu and intel
    • X: available with pgi, gnu, and intel
    * : Current Default Version

    You can use module spider hdf5  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    HDF5 is available without restriction to all OSC users.

    Usage on Oakley

    Set-up

    Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5 . To load a particular version, use  module load hdf5/version . For example, use  module load hdf5/1.8.15  to load HDF5 version 1.8.15. You can use module spider hdf5  to view available modules.

    Building With HDF5

    The HDF5 library provides the following variables for use at build time:

    Variable Use
    $HDF5_C_INCLUDE Use during your compilation step for C programs
    $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
    $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
    $HDF5_C_LIBS Use during your linking step programs
    $HDF5_F90_LIBS Use during your linking step for FORTRAN programs

     

    For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

    icc -c $HDF5_C_INCLUDE myprog.c
    icc -o myprog myprog.o $HDF5_C_LIBS
    ifort -c $HDF5_F90_INCLUDE myprog.f90
    ifort -o myprog myprog.o $HDF5_F90_LIBS
    

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Non-interactive Batch Job (Serial Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
    #PBS -N AppNameJob
    #PBS -l nodes=1:ppn=12
    module load hdf5
    cd $PBS_O_WORKDIR
    cp foo.dat $TMPDIR
    cd $TMPDIR
    appname
    cp foo_out.h5 $PBS_O_WORKDIR
    

    Usage on Ruby

    Set-up

    Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5 . To load a particular version, use  module load hdf5/version . For example, use  module load hdf5/1.8.15  to load HDF5 version 1.8.15. You can use module spider hdf5  to view available modules.

    Building With HDF5

    The HDF5 library provides the following variables for use at build time:

    Variable Use
    $HDF5_C_INCLUDE Use during your compilation step for C programs
    $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
    $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
    $HDF5_C_LIBS Use during your linking step programs
    $HDF5_F90_LIBS

    Use during your linking step for FORTRAN programs

     

    For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

    icc -c $HDF5_C_INCLUDE myprog.c
    icc -o myprog myprog.o $HDF5_C_LIBS
    ifort -c $HDF5_F90_INCLUDE myprog.f90
    ifort -o myprog myprog.o $HDF5_F90_LIBS
    

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Non-interactive Batch Job (Serial Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
    #PBS -N AppNameJob
    #PBS -l nodes=1:ppn=20
    module load hdf5
    cd $PBS_O_WORKDIR
    cp foo.dat $TMPDIR
    cd $TMPDIR
    appname
    cp foo_out.h5 $PBS_O_WORKDIR
    

    Usage on Owens

    Set-up

    Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library, run the following command: module load hdf5 . To load a particular version, use  module load hdf5/version . For example, use  module load hdf5/1.8.17  to load HDF5 version 1.8.17. You can use module spider hdf5  to view available modules.

    Building With HDF5

    The HDF5 library provides the following variables for use at build time:

    Variable Use
    $HDF5_C_INCLUDE Use during your compilation step for C programs
    $HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
    $HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
    $HDF5_C_LIBS Use during your linking step programs
    $HDF5_F90_LIBS

    Use during your linking step for FORTRAN programs

     

    For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

    icc -c $HDF5_C_INCLUDE myprog.c
    icc -o myprog myprog.o $HDF5_C_LIBS
    ifort -c $HDF5_F90_INCLUDE myprog.f90
    ifort -o myprog myprog.o $HDF5_F90_LIBS
    

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Non-interactive Batch Job (Serial Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
    #PBS -N AppNameJob
    #PBS -l nodes=1:ppn=28
    module load hdf5
    cd $PBS_O_WORKDIR
    cp foo.dat $TMPDIR
    cd $TMPDIR
    appname
    cp foo_out.h5 $PBS_O_WORKDIR
    

    Further Reading

    Tag: 
    Service: 

    HISAT2

    HISAT2 is a graph-based alignment program that maps DNA and RNA sequencing reads to a population of human genomes.

    Availability and Restrictions

    Versions

    HISAT2 is available on the Owens Cluster. The versions currently available at OSC are:

    VERSION OWENS
    2.1.0 X*
    * Current Default Version

    You can use module spider hisat2to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    HISAT2 is available without restriction to all OSC users.

    Usage

    Usage on Owens

    Set-up

    To configure your enviorment for use of HISAT2, use command module load hisat2. This will load the default version.

    Further Reading

    Fields of Science: 

    HMMER

    Profile hidden Markov models (profile HMMs) can be used to do sensitive database searching using statistical descriptions of a sequence family's consensus. HMMER uses profile HMMs, and can be useful in situations like:

    • if you are working with an evolutionarily diverse protein family, a BLAST search with any individual sequence may not find the rest of the sequences in the family.
    • the top hits in a BLAST search are hypothetical sequences from genome projects.
    • your protein consists of several domains which are of different types.

    HMMER (pronounced 'hammer', as in a more precise mining tool than BLAST) was developed by Sean Eddy at Washington University in St. Louis.

    HMMER is a very cpu-intensive program and is parallelized using threads, so that each instance of hmmsearch or the other search programs can use all the cpus available on a node. HMMER on OSC clusters are intended for those who need to run HMMER searches on large numbers of query sequences.

    Availability & Restrictions

    HMMER is available to all OSC users without restriction.

    The following versions of HMMER are available on OSC systems:

    Version Oakley
    3.0 X

    Usage

    Set-up

    To use HMMER on Oakley, first run the following command:

    module load hmmer

    Using HMMER

    Once the hmmer module is loaded, the following commands will be available for your use:

    Single sequence queries: new to HMMER3
    phmmer Search a sequence against a sequence database. (BLASTP-like)
    jackhmmer Iteratively search a sequence against a sequence database. (PSIBLAST-like)
    
    Replacements for HMMER2’s functionality
    hmmbuild Build a profile HMM from an input multiple alignment.
    hmmsearch Search a profile HMM against a sequence database.
    hmmscan Search a sequence against a profile HMM database.
    hmmalign Make a multiple alignment of many sequences to a common profile HMM.
    
    Other utilities
    hmmconvert Convert profile formats to/from HMMER3 format.
    hmmemit Generate (sample) sequences from a profile HMM.
    hmmfetch Get a profile HMM by name or accession from an HMM database.
    hmmpress Format an HMM database into a binary format for hmmscan.
    hmmstat Show summary statistics for each profile in an HMM database.
    

    If you need to know options for a command, type the command name followed by "-h", for example:

    hmmalign -h 

    Batch Usage

    A sample batch job is below:

    #PBS -N hmmer
    #PBS -j oe
    #PBS -l nodes=1:ppn=1
    #PBS -S /bin/bash
    
    cd $PBS_O_WORKDIR
    hmmalign globins4.align globins45
    hmmbuild globins4.hmm globins4.align
    hmmsearch globins4.hmm /fdb/fastadb/nr.aa.fas > globins.out

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    HOMER

    HOMER (Hypergeometric Optimization of Motif EnRichment) is a suite of tools for Motif Discovery and ChIP-Seq analysis. It is a collection of command line programs for unix-style operating systems written in mostly perl and c++. Homer was primarily written as a de novo motif discovery algorithm that is well suited for finding 8-12 bp motifs in large scale genomics data.

    Availability and Restrictions

    The following versions of HOMER are available on OSC clusters:

    Version Oakley owens
    4.8 X* X*
    *: Current default version

    You can use  module spider homer  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    HOMER is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of HOMER, run the following command:   module load homer. The default version will be loaded. To select a particular HOMER version, use module load homer/version. For example, use  module load homer/4.8 to load HOMER 4.8.

    Usage on Owens

    Set-up

    To configure your environment for use of HOMER, run the following command:   module load homer. The default version will be loaded. To select a particular HOMER version, use module load homer/version. For example, use module load homer/4.8 to load HOMER 4.8.

    Further Reading

    Service: 
    Fields of Science: 

    HTSlib

    HTSlib is a C library used for reading and writing high-throughput sequencing data. HTSlib is the core library used by SAMtools. HTSlib also provides the bgziphtsfile, and tabix utilities.

    Availability and Restrictions

    The versions of HTSlib currently available at OSC are:

    VERSION Oakley OWENS
    1.6 X* X*
    * Current Default Version

    Access

    HTSlib is available to all OSC users.

    Usage

    Usage on Oakley

    Set-up

    To set up the environment for HTSlib on the OSC clusters, use the command:

    module load htslib/1.6

    Usage on Owens

    Set-up

    To set up the environment for HTSlib on the OSC clusters, use the command:

    module load htslib/1.6

    Further Reading

    Hadoop

    A hadoop cluster can be launched within the HPC environment, but managed by the PBS job scheduler using  Myhadoop framework developed by San Diego Supercomputer Center. (Please see http://www.sdsc.edu/~allans/MyHadoop.pdf)

    Availability & Restrictions

    Hadoop is available to all OSC users without restriction.

    The following versions of Hadoop are available on OSC systems: 

    VERSION

    OAKLEY

    OWENS

    3.0.0*

     

    X

     

    NOTE: * means it is the default version.

    Set-up

    In order to configure your environment for the usage of Hadoop, run the following command:

    module load hadoop

    In order to access a particular version of Hadoop, run the following command

    module load hadoop/3.0.0-alpha1
    

    Using Hadoop

    In order to run Hadoop in batch, reference the example batch script below. This script requests 6 node on the Owens cluster for 1 hour of walltime. 

    #PBS -N hadoop-example
    
    #PBS -l nodes=6:ppn=12
    
    #PBS -l walltime=01:00:00
    
    setenv WORK $PBS_O_WORKDIR
    
    module load hadoop/3.0.0-alpha1
    
    module load myhadoop/v0.40
    
    setenv HADOOP_CONF_DIR $TMPDIR/mycluster-conf-$PBS_JOBID
    
    cd $TMPDIR
    
    myhadoop-configure.sh -c $HADOOP_CONF_DIR -s $TMPDIR
    
    $HADOOP_HOME/sbin/start-dfs.sh
    
    hadoop dfsadmin -report
    
    hadoop  dfs -mkdir data
    
    hadoop  dfs -put $HADOOP_HOME/README.txt  data/
    
    hadoop  dfs -ls data
    
    hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1.jar wordcount data/README.txt wordcount-out
    
    hadoop  dfs -ls wordcount-out
    
    hadoop  dfs  -copyToLocal -f  wordcount-out  $WORK
    
    $HADOOP_HOME/sbin/stop-dfs.sh
    
    myhadoop-cleanup.sh
    
    

    Example Jobs

    Please check /usr/local/src/hadoop/3.0.0-alpha1/test.osc folder for more examples of hadoop jobs

    Further Reading

    See Also

    Service: 
    Fields of Science: 

    Intel Compilers

    The Intel compilers for both C/C++ and FORTRAN.

    Availability and Restrictions

    The versions currently available at OSC are:

    Version Oakley Ruby owens notes
    12.1.4.319 X     Default version on Oakley prior to 09/15/2015
    13.1.3.192 X      
    14.0.0.080 X      
    14.0.3   X    
    15.0.0   X   Default version on Ruby prior to 09/15/2015
    15.0.1   X    
    15.0.3 X* X*    
    16.0.3 X X X*  
    * : Current Default Version

    You can use module avail intel  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    The Intel Compilers are available to all OSC users without restriction.

    Usage

    Usage on Oakley

    Set-up on Oakley

    After you ssh to Oakley, the default version of Intel compilers will be loaded for you automatically. To configure your environment for a particular Intel compilers version, use  module swap intel/12.1.4.39 intel/version . For example, use  module swap intel/12.1.4.39 intel/14.0.0.080  to load Intel  compilers 14.0.0.080 on Oakley.

    Using the Intel Compilers

    Once the intel compiler module has been loaded, the compilers are available for your use. The following table lists common compiler options available in all languages.

    Compiler option Purpose
    -c Compile only; do not link
    -DMACRO[=value] Defines preprocessor macro MACRO with optional value (default value is 1)
    -g Enables debugging; disables optimization
    -I/directory/name Add /directory/name to the list of directories to be searched for #include files
    -L/directory/name Adds /directory/name to the list of directories to be searched for library files
    -lname Adds the library libname.a or libname.so to the list of libraries to be linked
    -o outfile Names the resulting executable outfile instead of a.out
    -UMACRO Removes definition of MACRO from preprocessor
    -O0 Disable optimization
    -O1 Light optimization
    -O2 Heavy optimization (default)
    -O3 Aggressive optimization; may change numerical results
    -ipo Inline function expansion for calls to procedures defined in separate files
    -funroll-loops Loop unrolling
    -parallel Automatic parallelization
    -openmp Enables translation of OpenMP directives

    The following table lists some options specific to C/C++

    -strict-ansi Enforces strict ANSI C/C++ compliance
    -ansi Enforces loose ANSI C/C++ compliance

    The following table lists some options specific to Fortran

    -convert big_endian Use unformatted I/O compatible with Sun and SGI systems
    -convert cray Use unformatted I/O compatible with Cray systems
    -i8 Makes 8-byte INTEGERs the default
    -module /dir/name Adds /dir/name to the list of directories searched for Fortran 90 modules
    -r8 Makes 8-byte REALs the default
    -fp-model strict Disables optimizations that can change the results of floating point calculations

    Some parts of the C++11 and C++14 standards are available by default in various versions of the Intel compilers; others require you to load an extra module. 

    With the Intel 14 or 15 compilers:

    module load cxx11
    

    With the Intel 16 compiler:

    module load cxx14
    

    See our compilation guide for suggestions on how to compile your software on our systems.

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session
    For an interactive batch session on Oakley, one can run the following command:
    qsub -I -l nodes=1:ppn=12 -l walltime=1:00:00
    
    which gives you 12 cores ( -l nodes=1:ppn=12 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named hello.c  and the output file named hello_results . Below is the example batch script ( job.txt ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=12
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    cp hello.c $TMPDIR
    cd $TMPDIR
    icc -O2 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script ( job.txt ) for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    mpicc -O2 hello.c -o hello
    pbsdcp hello $TMPDIR
    cd $TMPDIR
    mpiexec hello > hello_results
    pbsdcp -g hello_results $PBS_O_WORKDIR
    

    Usage on Ruby

    Set-up on Ruby

    After you ssh to Ruby, the default version of Intel compilers will be loaded for you automatically. To configure your environment for a particular Intel compilers version, use module swap intel/15.0.0 intel/version . For example, use  module swap intel/15.0.0 intel/15.0.1  to load Intel  compilers 15.0.1 on Ruby.

    Using the Intel Compilers

    Once the intel compiler module has been loaded, the compilers are available for your use. The following table lists common compiler options available in all languages.

    Compiler option Purpose
    -c Compile only; do not link
    -DMACRO[=value] Defines preprocessor macro MACRO with optional value (default value is 1)
    -g Enables debugging; disables optimization
    -I/directory/name Add /directory/name to the list of directories to be searched for #include files
    -L/directory/name Adds /directory/name to the list of directories to be searched for library files
    -lname Adds the library libname.a or libname.so to the list of libraries to be linked
    -o outfile Names the resulting executable outfile instead of a.out
    -UMACRO Removes definition of MACRO from preprocessor
    -O0 Disable optimization
    -O1 Light optimization
    -O2 Heavy optimization (default)
    -O3 Aggressive optimization; may change numerical results
    -ipo Inline function expansion for calls to procedures defined in separate files
    -funroll-loops Loop unrolling
    -parallel Automatic parallelization
    -openmp Enables translation of OpenMP directives

    The following table lists some options specific to C/C++

    -strict-ansi Enforces strict ANSI C/C++ compliance
    -ansi Enforces loose ANSI C/C++ compliance

    The following table lists some options specific to Fortran

    -convert big_endian Use unformatted I/O compatible with Sun and SGI systems
    -convert cray Use unformatted I/O compatible with Cray systems
    -i8 Makes 8-byte INTEGERs the default
    -module /dir/name Adds /dir/name to the list of directories searched for Fortran 90 modules
    -r8 Makes 8-byte REALs the default
    -fp-model strict Disables optimizations that can change the results of floating point calculations

    Some parts of the C++11 and C++14 standards are available by default in various versions of the Intel compilers; others require you to load an extra module. 

    With the Intel 14 or 15 compilers:

    module load cxx11
    

    With the Intel 16 compiler:

    module load cxx14
    

    See our compilation guide for suggestions on how to compile your software on our systems.

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Ruby, one can run the following command:
    qsub -I -l nodes=1:ppn=20 -l walltime=1:00:00
    
    which gives you 20 cores ( -l nodes=1:ppn=20 ) with 1 hour ( -l walltime=1:00:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named  hello.c  and the output file named  hello_results . Below is the example batch script ( job.txt ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=20
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    cp hello.c $TMPDIR
    cd $TMPDIR
    icc -O2 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the job.txt  file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script (  job.txt   for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=20
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    mpicc -O2 hello.c -o hello
    cp hello $TMPDIR
    cd $TMPDIR
    mpiexec ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR​
    

    Usage on Owens

    Set-up on Owens

    After you ssh to Owens, the default version of Intel compilers will be loaded for you automatically. 

    Using the Intel Compilers

    Once the intel compiler module has been loaded, the compilers are available for your use. The following table lists common compiler options available in all languages.

    COMPILER OPTION PURPOSE
    -c  Compile only; do not link
    -DMACRO[=value]  Defines preprocessor macro MACRO with optional value (default value is 1)
    -g  Enables debugging; disables optimization
    -I/directory/name  Add /directory/name to the list of directories to be searched for #include files
    -L/directory/name  Adds /directory/name to the list of directories to be searched for library files
    -lname Adds the library libname.a or libname.so to the list of libraries to be linked
    -o outfile  Names the resulting executable outfile instead of a.out
    -UMACRO  Removes definition of MACRO from preprocessor
    -O0  Disable optimization
    -O1  Light optimization
    -O2  Heavy optimization (default)
    -O3  Aggressive optimization; may change numerical results
    -ipo  Inline function expansion for calls to procedures defined in separate files
    -funroll-loops  Loop unrolling
    -parallel  Automatic parallelization
    -openmp  Enables translation of OpenMP directives

    The following table lists some options specific to C/C++

    -strict-ansi  Enforces strict ANSI C/C++ compliance
    -ansi  Enforces loose ANSI C/C++ compliance

    The following table lists some options specific to Fortran

    -convert big_endian  Use unformatted I/O compatible with Sun and SGI systems
    -convert cray  Use unformatted I/O compatible with Cray systems
    -i8  Makes 8-byte INTEGERs the default
    -module /dir/name  Adds /dir/name to the list of directories searched for Fortran 90 modules
    -r8  Makes 8-byte REALs the default
    -fp-model strict  Disables optimizations that can change the results of floating point calculations

    Some parts of the C++11 standard are available by default in various versions of the Intel compilers; others require you to load an extra module. 

    With the Intel 14 or 15 compilers:

    module load cxx11
    

    See our compilation guide for suggestions on how to compile your software on our systems.

    Batch Usage on Owens

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session on Owens, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores (  -l nodes=1:ppn=28  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)
    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will use the input file named   hello.c   and the output file named   hello_results  . Below is the example batch script (  job.txt  ) for a serial run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=1:ppn=28
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    cp hello.c $TMPDIR
    cd $TMPDIR
    icc -O2 hello.c -o hello
    ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR
    

    In order to run it via the batch system, submit the  job.txt   file with the following command:

    qsub job.txt
    
    Non-interactive Batch Job (Parallel Run)
    Below is the example batch script (   job.txt    for a parallel run:
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=28
    #PBS -N jobname
    #PBS -j oe
    cd $PBS_O_WORKDIR
    module load intel
    mpicc -O2 hello.c -o hello
    cp hello $TMPDIR
    cd $TMPDIR
    mpiexec ./hello > hello_results
    cp hello_results $PBS_O_WORKDIR​
    
     

    Further Reading

    See Also

    Service: 
    Technologies: 
    Fields of Science: 

    Intel MPI

    https://www.osc.edu/supercomputing/software/intel-compilers Intel's implementation of the Message Passing Interface (MPI) library.

    Availability and Restrictions

    Intel MPI may be used as an alternative to - but not in conjunction with - the MVAPICH2 MPI libraries. The versions currently available at OSC are:

    Version Oakley ruby owens notes
    4.0.3.008 X     Default version on Oakley prior to 09/15/2015
    4.1.0.024 X      
    4.1.1.036 X      
    4.1.2.040 X      
    4.1.3 X X    
    4.1.3.049 X X    
    5.0.1   X    
    5.0.1.035   X    
    5.0.3 X* X*    
    5.1.3     X*  
    * : Current Default Version

    You can use module spider intelmpi  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Intel MPI is available to all OSC users without restriction.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To configure your environment for the default version of Intel MPI, use  module load intelmpi . To configure your environment for a specific version of Intel MPI, use  module load intelmpi/version . For example, use  module load intelmpi/4.1.3.049  to load Intel MPI version 4.1.3.049 on Oakley.

    You can use module spider intelmpi  to view available modules on Oakley.

    Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi .

    Using Intel MPI

    Software compiled against this module will use the libraries at runtime.

    Building With Intel MPI

    On Oakley, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

    Variable Use
    $MPI_CFLAGS Use during your compilation step for C programs.
    $MPI_CXXFLAGS Use during your compilation step for C++ programs.
    $MPI_FFLAGS Use during your compilation step for Fortran programs.
    $MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
    $MPI_LIBS Use when linking your program to Intel MPI.

    In general, for any application already set up to use mpicc , (or similar), compilation should be fairly straightfoward. 

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Non-interactive Batch Job (Parallel Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application ) for five-hours on Oakley:
    #PBS -N MyIntelMPIJob
    #PBS -l nodes=4:ppn=12
    #PBS -l walltime=5:00:00
    module swap mvapich2 intelmpi
    cd $PBS_O_WORKDIR
    mpiexec my-impi-application
    

    Usage on Ruby

    Set-up on Ruby

    To configure your environment for the default version of Intel MPI, use  module load intelmpi . To configure your environment for a specific version of Intel MPI, use  module load intelmpi/version . For example, use  module load intelmpi/4.1.3.049  to load Intel MPI version 4.1.3.049 on Ruby.

    You can use module spider intelmpi  to view available modules on Ruby.

    Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use module load intelmpi .

    Using Intel MPI

    Software compiled against this module will use the libraries at runtime.

    Building With Intel MPI

    On Ruby, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

    Variable Use
    $MPI_CFLAGS Use during your compilation step for C programs.
    $MPI_CXXFLAGS Use during your compilation step for C++ programs.
    $MPI_FFLAGS Use during your compilation step for Fortran programs.
    $MPI_F90FLAGS Use during your compilation step for Fortran 90 programs.
    $MPI_LIBS Use when linking your program to Intel MPI.

    In general, for any application already set up to use mpicc , (or similar), compilation should be fairly straightfoward. 

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Non-interactive Batch Job (Parallel Run)
    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called my-impi-application ) for five-hours on Ruby:
    #PBS -N MyIntelMPIJob
    #PBS -l nodes=4:ppn=20
    #PBS -l walltime=5:00:00
    module swap mvapich2 intelmpi
    cd $PBS_O_WORKDIR
    mpiexec my-impi-application
    

    Usage on Owens

    Set-up on Owens

    To configure your environment for the default version of Intel MPI, use   module load intelmpi  . To configure your environment for a specific version of Intel MPI, use   module load intelmpi/version  . For example, use   module load intelmpi/5.1.3   to load Intel MPI version 5.1.3 on Owens.

    You can use  module spider intelmpi   to view available modules on Owens.

    Note: This module conflicts with the default loaded MVAPICH installations, and Lmod will automatically replace with the correct one when you use  module load intelmpi  .

    Using Intel MPI

    Software compiled against this module will use the libraries at runtime.

    Building With Intel MPI

    On Ruby, we have defined several environment variables to make it easier to build and link with the Intel MPI libraries.

    VARIABLE USE
    $MPI_CFLAGS  Use during your compilation step for C programs.
    $MPI_CXXFLAGS  Use during your compilation step for C++ programs.
    $MPI_FFLAGS  Use during your compilation step for Fortran programs.
    $MPI_F90FLAGS  Use during your compilation step for Fortran 90 programs.
    $MPI_LIBS  Use when linking your program to Intel MPI.

    In general, for any application already set up to use  mpicc  , (or similar), compilation should be fairly straightfoward. 

    Batch Usage on Owens

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Non-interactive Batch Job (Parallel Run)
    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. The following example batch script file will run a program compiled against Intel MPI (called  my-impi-application  ) for five-hours on Ruby:
    #PBS -N MyIntelMPIJob
    #PBS -l nodes=4:ppn=28
    #PBS -l walltime=5:00:00
    module swap mvapich2 intelmpi
    cd $PBS_O_WORKDIR
    mpiexec my-impi-application
    
     

    Further Reading

    See Also

    Service: 
    Technologies: 
    Fields of Science: 

    JasPer

    Introduction

    JasPer is an extensible open source utility designed for the manipulation, conversion, compression and decompression of digital images.  Currently supported image formats include bmp, jp2, jpc, jpg, pgx, mif and ras.    

    Availability

    Version 1.900.1 is available on the Oakley cluster.  

    Usage

    In order to use the JasPer library, you must load the JasPer module into your environment.  To load the JasPer module, type the following command:

    module load jasper

    Useful commands

    In addition to the JasPer libraries, a few default applications are also provided when the JasPer module is loaded.     

    To convert an image from one format to another, use the jasper command.   

    jasper [options]

    For a comprehensive list of options available for this command, type

    jasper --help

    To compare particular qualities of two images, use the imgcmp command.

    imgcmp -f [file 1] -F [file 2] -m [metric]

    For a full list of acceptable metric arguments, type

    imgcmp --help

    To view a sequence of one or more images, use the jiv command.  

    jiv image

    Alternatively, if you would like to view multiple images in sequence you can use the --wait option to specify the amount of time between the display of each image, creating a slideshow.  The command below will display each image for 5 seconds before displaying the next image.   

    jiv --wait 5 [image1 image2 image3 ...]

    For a full list of options, type

    jiv --help

    Further Reading

    Additional information about JasPer can be found online at the JasPer Project Homepage.

    Service: 

    Julia

    From julialang.org:

    "Julia is a high-level, high-performance dynamic programming language for numerical computing. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing. In addition, the Julia developer community is contributing a number of external packages through Julia’s built-in package manager at a rapid pace. IJulia, a collaboration between the Jupyter and Julia communities, provides a powerful browser-based graphical notebook interface to Julia."

    Interactive Julia Notebooks

    If you are using OnDemand, you can simply work with Jupyter and the selection of the Julia kernel to use interactive notebooks to work on an Owens compute node!

    Navigate to ondemand.osc.edu and select a Jupyter notebook:

    Jupyter Notebook

    Availability and Restrictions

    Julia is available on Owens Clusters. The versions currently available at OSC are:

    Version Oakley ruby owens notes
    0.5.1     X* This version works with Jupyter notebooks
    *: Current default version

    Feel free to contact OSC Help if you need other versions for your work.

    Access

    Julia is available for use by all OSC users.

    Service: 
    Fields of Science: 

    Kallisto

    Kallisto is an RNA-seq quantification program. It quantifies abundances of transcripts from RNA-seq data and uses psedoalignment to determine the compatibility of reads with targets, without needing alignment.

    Availability and Restrictions

    Kallisto is available on the Owens Clusters. The versions currently available at OSC are:

    VERSION OWENS
    0.43.1 X*
    * Current Default Version

    You can use module spider kallisto to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    Kallisto is available without restriction to all OSC users.

    Usage

    Usage on Owens

    Set-up

    To configure your enviorment for use of Salmon, use command module load kallisto. This will load the default version.

    Further Reading

    Fields of Science: 

    LAMMPS

    The Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a classical molecular dynamics code designed for high-performance simulation of large atomistic systems.  LAMMPS generally scales well on OSC platforms, provides a variety of modeling techniques, and offers GPU accelerated computation.

    Availability and Restrictions

    LAMMPS is available on Oakley, Ruby, and Owens Clusters. The following versions are currently available at OSC (S means serial executables, P means parallel, and C means CUDA, i.e., GPU enabled):

    Version Oakley Ruby Owens notes
    Feb12 PC     Default version on Oakley prior to 09/15/2015
    May12 PC      
    Feb14 SPC      
    5Sep14 P P    
    7Dec15 PC PC    
    14May16 PC* PC* P*  
    31Mar17     PC  
    *: Current default version

    You can use module spider lammps to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    LAMMPS is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To load the default version of LAMMPS module and set up your environment, use  module load lammps . To select a particular software version, use   module load lammps/version . For example, use  module load lammps/12Feb12   to load LAMMPS version Feb12. 

    Using LAMMPS

    Once a module is loaded, LAMMPS can be run with the following command:
    lammps < input.file
    

    To see information on the packages and executables for a particular installation, run the module help command, for example:

    module help lammps
    

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=1 -l walltime=00:20:00
    

    which requests one core ( -l nodes=1:ppn=1 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Non-interactive Batch Job (Parallel Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

    ~srb/workshops/compchem/lammps/
    

    Below is a sample batch script. It asks for 24 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

    #PBS -N chain  
    #PBS -l nodes=2:ppn=12  
    #PBS -l walltime=10:00:00  
    #PBS -S /bin/bash  
    #PBS -j oe  
    module load lammps  
    cd $PBS_O_WORKDIR  
    pbsdcp chain.in $TMPDIR  
    cd $TMPDIR  
    lammps < chain.in  
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Usage on Ruby

    Set-up

    To load the default version of LAMMPS module and set up your environment, use  module load lammps . To select a particular software version, use   module load lammps/version . For example, use  module load lammps/5Sep14   to load LAMMPS version 5Sep14. 

    Using LAMMPS

    Once a module is loaded, LAMMPS can be run with the following command:
    lammps < input.file
    

    To see information on the packages and executables for a particular installation, run the module help command, for example:

    module help lammps
    

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=20:gpus=1 -l walltime=00:20:00 
    

    which requests one whole node with 20 cores ( -l nodes=1:ppn=20 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Non-interactive Batch Job (Parallel Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

    ~srb/workshops/compchem/lammps/
    

    Below is a sample batch script. It asks for 40 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

    #PBS -N chain  
    #PBS -l nodes=2:ppn=20  
    #PBS -l walltime=10:00:00  
    #PBS -S /bin/bash  
    #PBS -j oe  
    module load lammps  
    cd $PBS_O_WORKDIR  
    pbsdcp chain.in $TMPDIR  
    cd $TMPDIR  
    lammps < chain.in  
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Usage on Owens

    Set-up

    To load the default version of LAMMPS module and set up your environment, use  module load lammps . To select a particular software version, use   module load lammps/version . For example, use  module load lammps/14May16    to load LAMMPS version 14May16. 

    Using LAMMPS

    Once a module is loaded, LAMMPS can be run with the following command:
    lammps < input.file
    

    To see information on the packages and executables for a particular installation, run the module help command, for example:

    module help lammps
    

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=28:gpus=1 -l walltime=00:20:00 
    

    which requests one whole node with 28 cores ( -l nodes=1:ppn=28  ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Non-interactive Batch Job (Parallel Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Sample batch scripts and LAMMPS input files are available here:

    ~srb/workshops/compchem/lammps/
    

    Below is a sample batch script. It asks for 56 processors and 10 hours of walltime. If the job goes beyond 10 hours, the job would be terminated.

    #PBS -N chain  
    #PBS -l nodes=2:ppn=28  
    #PBS -l walltime=10:00:00  
    #PBS -S /bin/bash  
    #PBS -j oe  
    module load lammps  
    cd $PBS_O_WORKDIR  
    pbsdcp chain.in $TMPDIR  
    cd $TMPDIR  
    lammps < chain.in  
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Further Reading

    Service: 

    LAPACK

    LAPACK (Linear Algebra PACKage) provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.

    Availability and Restrictions

    A highly optimized implementation of LAPACK is available on all OSC clusters as part of the Intel Math Kernel Library (MKL). We recommend that you use MKL rather than building LAPACK for yourself. MKL is available to all OSC users.

    Usage

    See OSC's MKL software page for usage information. Note that there is no library named liblapack.a or liblapack.so. The flag "-llapack" on your link line will not work. You should modify your makefile or build script to link to the MKL libraries instead.

    Further Reading

    Service: 
    Technologies: 
    Fields of Science: 

    LS-DYNA

    LS-DYNA is a general purpose finite element code for simulating complex structural problems, specializing in nonlinear, transient dynamic problems using explicit integration. LS-DYNA is one of the codes developed at Livermore Software Technology Corporation (LSTC).

    Availability and Restrictions

    LS-DYNA is available on Ruby and Oakley Clusters for both serial (smp solver for single node jobs) and parallel (mpp solver for multipe node jobs) versions. The versions currently available at OSC are:

    Version Oakley Ruby Owens notes
    971-R4.2.1 smp         
    mpp         
    971-R5 smp         
    mpp         
    971-R5.1.1 smp X     Default version on Oakley prior to 09/15/2015
    mpp X     Default version on Oakley prior to 09/15/2015
    971-R7.0.0 smp X X*    
    mpp X X*    

    971-R7.1.1

    smp  X*      
    mpp X*      

    971-R9.0.1

    smp  X   X*  
    mpp X   X*  
    *: Current default version

    Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    OSC does not provide LS_DYNA license directly, however users with their own academic departmental license can use it on the OSC clusters.  Please contact OSC Help for further instruction.

    Access for Commerical Users

    Contact OSC Help for getting access to LS-DYNA if you are a commerical user.

    Usage

    Usage on Oakley

    Set-up on Oakley

    To view available modules installed on Oakley, use  module spider ls-dyna  for smp solvers, and use  module spider mpp  for mpp solvers. In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp-dyna/971_d_R7.0.0 is the mpp solver with double precision on Oakley. Use  module load name  to load LS-DYNA with a particular software version. For example, use  module load mpp-dyna/971_d_R7.0.0   to load LS-DYNA mpp solver version 7.0.0 with double precision on Oakley.

    Batch Usage on Oakley

    When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=12 -l walltime=00:20:00 
    
    which requests one whole node with 12 cores ( -l nodes=1:ppn=12 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files ( explorer.k  in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as job.txt . It uses the smp solver for a serial job (nodes=1) on Oakley:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=1:ppn=12
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module load ls-dyna/971_d_7.0.0
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    lsdyna I=explorer.k NCPU=12
    

     3) submit the script to the batch queue with the command:  qsub job.txt

     When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. 

    Non-interactive Batch Job (Parallel Run)
    Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files ( explorer.k  in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as job.txt ). It uses the mmp solver for a parallel job (nodes>1) on Oakley:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=2:ppn=12
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module swap mvapich2/1.7 intelmpi
    module load mpp-dyna/971_d_R7.0.0
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    mpiexec mpp971 I=explorer.k NCPU=24
    

     3) submit the script to the batch queue with the command:  qsub job.txt

    When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

    ...
    cd $TMPDIR
    cp $PBS_O_WORKDIR/explorer.k .
    ... #launch the solver and execute
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Usage on Ruby

    Set-up on Ruby

    To view available modules installed on Ruby, use  module spider ls-dyna  for smp solvers, and use  module spider mpp  for mpp solvers. In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp-dyna/971_d_R7.0.0 is the mpp solver with double precision on Ruby. Use  module load name  to load LS-DYNA with a particular software version. For example, use  module load mpp-dyna/971_d_R7.0.0   to load LS-DYNA mpp solver version 7.0.0 with double precision on Ruby.

    Batch Usage on Ruby

    When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=20 -l walltime=00:20:00 
    

    which requests one whole node with 20 cores ( -l nodes=1:ppn=20 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). You may adjust the numbers per your need.

    Non-interactive Batch Job (Serial Run)

    A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files ( explorer.k  in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as job.txt . It uses the smp solver for a serial job (nodes=1) on Ruby:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=1:ppn=20
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module load ls-dyna/971_d_7.0.0
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    lsdyna I=explorer.k NCPU=20
    

     3) submit the script to the batch queue with the command:  qsub job.txt

     When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. 

    Non-interactive Batch Job (Parallel Run)

    Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files ( explorer.k  in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as job.txt ). It uses the mmp solver for a parallel job (nodes>1) on Ruby:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=2:ppn=20
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module swap mvapich2/1.7 intelmpi/5.0.1
    module load mpp-dyna/971_d_R7.0.0
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    mpiexec mpp971 I=explorer.k NCPU=40
    

     3) submit the script to the batch queue with the command:  qsub job.txt

    When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

    ...
    cd $TMPDIR
    cp $PBS_O_WORKDIR/explorer.k .
    ... #launch the solver and execute
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Usage on Owens

    Set-up on Owens

    To view available modules installed on Owens, use   module spider ls-dyna   for smp solvers, and use  module spider mpp   for mpp solvers. In the module name, '_s' indicates single precision and '_d' indicates double precision. For example, mpp-dyna/971_d_R9.0.1 is the mpp solver with double precision on Owens. Use  module load name   to load LS-DYNA with a particular software version. For example, use  module load mpp-dyna/971_d_R9.0.1    to load LS-DYNA mpp solver version 9.0.1 with double precision on Owens.

    Batch Usage on Owens

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session

    For an interactive batch session one can run the following command:

    qsub -I -l nodes=1:ppn=28 -l walltime=00:20:00 
    
    which requests one whole node with 28 cores (  -l nodes=1:ppn=28  ), for a walltime of 20 minutes ( -l walltime=00:20:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files (  explorer.k   in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as  job.txt  . It uses the smp solver for a serial job (nodes=1) on Owens:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=1:ppn=28
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module load ls-dyna/971_d_9.0.1
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    lsdyna I=explorer.k NCPU=28
    

     3) submit the script to the batch queue with the command:   qsub job.txt

     When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. 

    Non-interactive Batch Job (Parallel Run)
    Please follow the steps below to use LS-DYNA via the batch system:

    1) copy your input files (  explorer.k   in the example below) to your work directory at OSC

    2) create a batch script, similar to the following file, saved as  job.txt  ). It uses the mmp solver for a parallel job (nodes>1) on Owens:

    #PBS -N plate_test
    #PBS -l walltime=5:00:00
    #PBS -l nodes=2:ppn=28
    #PBS -j oe
    # The following lines set up the LSDYNA environment
    module swap mvapich2/2.2 intelmpi
    module load mpp-dyna/971_d_R9.0.1
    #
    # Move to the directory where the input files are located
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA (number of cpus > 1)
    #
    mpiexec mpp971 I=explorer.k NCPU=56
    

     3) submit the script to the batch queue with the command:   qsub job.txt

    When the job is finished, all the result files will be found in the directory where you submitted your job ($PBS_O_WORKDIR). Alternatively, you can submit your job from the temporary directory ($TMPDIR), which is faster to access for the system and might be beneficial for bigger jobs. Note that $TMPDIR is uniquely associated with the job submitted and will be cleared when the job ends. So you need to copy your results back to your work directory at the end of your script. An example scrip should include the following lines:

    ...
    cd $TMPDIR
    cp $PBS_O_WORKDIR/explorer.k .
    ... #launch the solver and execute
    pbsdcp -g '*' $PBS_O_WORKDIR
    

    Further Reading

    See Also

    Service: 

    LS-PrePost

    Introduction

    LS-PrePost is an advanced pre and post-processor that is delivered free with LS-DYNA. The user interface is designed to be both efficient and intuitive. LS-PrePost runs on Windows, Linux, and Unix utilizing OpenGL graphics to achieve fast rendering and XY plotting. The latest builds can be downloaded from LSTC's FTP Site.

     

    The prefered way of accessing LS-Prepost is through OnDemand's Glenn desktop application.  This gives you a preconfigured enviornment with GPU acceleration enabled.

    Availability

    LS-PrePost is available on both Glenn and Oakley clusters. A module was created for LS-Prepost 4.0 on Oakley, and can be loaded with 'module load lspp'. All other versions can be loaded with the corresponding dyna modules.

    Version Oakley
    v3.0 X
    v3.2 X
    v4.0 X

     

     

     

    Usage

    Running LS-PrePost on Oakley through OnDemand's Glenn Destkop

    Below are instructions on running LS-PrePost on Oakley through Glenn's OnDemand desktop interface with GPU acceleration enabled.  To run LS-PrePost with a slower X-tunneling procedure, see the specified instructions below.

     

    1) Log in to OnDemand with your HPC username and password.

    2) Launch the Glenn Desktop from "Apps" menu. 

    3) Open a Terminal window (Applications > Accessories > Terminal)

    4) Type the following command to connect to oakley:

        ssh -X username@oakley.osc.edu       

              * Where "username" is your username.

    5)  Once logged in to Oakley, submit an interactive job with the following command:

        qsub -X -I -l nodes=1:ppn=12:gpus=2:vis -l walltime=hh:mm:ss

              * pick a walltime that is close to the amount of time you will need to for working in the GUI application.

    6) Once your job starts, make a note of the hostname for the compute node your job is running on.  You can find this information by typing the following command:

        hostname

    7) Open another Terminal window, and type the following commands:

         module load VirtualGL
         vglconnect username@job-hostname

               * job-hostname is the information you found in step 6; your command might look something like this, for example:  

         vglconnect ahahn@n0656.ten.osc.edu

               You'll be asked a password to connect, which should be your HPC password.

    8) Now, you should be connected to your running job's GPU node.  Run the following commands to launch LS-PrePost version 4.0:

         export LD_LIBRARY_PATH=/usr/local/MATLAB/R2013a/sys/opengl/lib/glnxa64
         /usr/local/lstc/ls-prepost/lsprepost4.0_centos6/lspp4

    At startup LS-PrePost displays a graphical interface for model generation and results post-processing.

     

    Running LS-PrePost on Oakley or Glenn with X-11 forwarding

    The following procedure will result in a much slower and GUI interface, but may be useful if the above instructions are not working.  It can be done completely from the command line, with no need for logging into OnDemand.  You may need to edit your terminal settings to enabled x11 forwarding.

    1) Login to Oakley or Glenn with X11 forwarding enabled

    ssh -X username@oakley.osc.edu

    or

    ssh -X username@glenn.osc.edu

     

    2) Submit a Interactive job

    qsub -I -X -l nodes=1:ppn=12 -l walltime=hh:mm:ss

    and wait for it to start. Use ppn=8 for Glenn.

     

    3) Load the LS-Dyna module and start up LS-PrePost application

    module load ls-dyna
    lspp3

     

       For LS-Prepost 4.0 on Oakley, use the following commands instead:

    module load lspp
    lspp4

     

    A x11 window should pop up.  If you get a error along the lines of:

    Error: Unable to initialize gtk, is DISPLAY set properly?

     

    Double check that you:

    1) logged in with x11 forwarding enabled

    2) have configured your x11 settings for your terminal

    3) included the -X with the qsub command

    4) have a x11 client running on your computer (Xming, Xorg, XQuarts, etc.).

    Documentation

    Documentation for LS-PrePost may be obtained at: http://www.lstc.com/lspp/

    User-Defined Material for LS-DYNA

    This page describes how to specify user defined material to use within LS-DYNA.  The user-defined subroutines in LS-DYNA allow the program to be customized for particular applications.  In order to define user material, LS-DYNA must be recompiled.

    Usage

    The first step to running a simulation with user defined material is to build a new executable. The following is an example done with solver version mpp971_s_R7.1.1.

    When you log into the Oakley system, load mpp971_s_R7.1.1 with the command:

    module load mpp-dyna/R7.1.1

    Next, copy the mpp971_s_R7.1.1 object files and Makefile to your current directory:

    cp /usr/local/lstc/mpp-dyna/R7.1.1/usermat/* $PWD

    Next, update the dyn21.f file with your user defined material model subroutine. Please see the LS-DYNA User's Manual (Keyword version) for details regarding the format and structure of this file.

    Once your user defined model is setup correctly in dyn21.f, build the new mpp971 executable with the command:

    make

    To execute a multi processor (ppn > 1) run with your new executable, execute the following steps:

    1) move your input file to a directory on an OSC system (pipe.k in the example below)

    2) copy your newly created mpp971 executable to this directory as well

    3) create a batch script (lstc_umat.job) like the following:

    #PBS -N LSDYNA_umat
    #PBS -l walltime=1:00:00
    #PBS -l nodes=2:ppn=8
    #PBS -j oe
    #PBS -S /bin/csh
    
    # This is the template batch script for running a pre-compiled
    # MPP 971 v7600 LS-DYNA.  
    # Total number of processors is ( nodes x ppn )
    #
    # The following lines set up the LSDYNA environment
    module load mpp-dyna/R7.1.1
    #
    # Move to the directory where the job was submitted from
    # (i.e. PBS_O_WORKDIR = directory where you typed qsub)
    #
    cd $PBS_O_WORKDIR
    #
    # Run LSDYNA 
    # NOTE: you have to put in your input file name
    #
    mpiexec mpp971 I=pipe.k NCPU=16

              4) Next, submit this job to the batch queue with the command:

           qsub lstc_umat.job

    The output result files will be saved to the directory you ran the qsub command from (known as the $PBS_O_WORKDIR_

    Documentation

    On-line documentation is available on LSTC website.

    See Also

     

     

    Service: 

    MAGMA

    MAGMA is a collection of next generation linear algebra (LA) GPU accelerated libraries designed and implemented by the team that developed LAPACK and ScaLAPACK. MAGMA is for heterogeneous GPU-based architectures, it supports interfaces to current LA packages and standards, e.g., LAPACK and BLAS, to allow computational scientists to effortlessly port any LA-relying software components. The main benefits of using MAGMA are that it can enable applications to fully exploit the power of current heterogeneous systems of multi/manycore CPUs and multi-GPUs, and deliver the fastest possible time to an accurate solution within given energy constraints.

    Availability and Restrictions

    MAGMA is available on Owens, and the following versions are currently available at OSC:

    Version Owens
    2.2.0 X(I)
    *: Current default version

    You can use module spider magma  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access for Academic Users

    MAGMA is available without restriction to all OSC users. 

    Usage on Owens

    Set-up

    To load the default version of MAGMA module, cuda must first be loaded. Use module load cuda to load the default version of cuda, or module load cuda/version to load a specific version. Then use  module load magma  to load MAGMA. To select a particular software version, use    module load magma/version  . For example, use   module load magma/2.2.0     to load MAGMA version 2.2.0

    Using MAGMA

    To run MAGMA in the command line, use the Intel compilers (icc, ifort). 

    icc $MAGMA_CFLAGS example.c
    

    or

    ifort $MAGMA_F90FLAGS example.F90
    

     

     

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your MAGMA simulation to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session, one can run the following command:
    qsub -I -l nodes=1:ppn=28 -l walltime=1:00:00
    
    which gives you 28 cores (  -l nodes=1:ppn=28  ) with 1 hour (  -l walltime=1:00:00  ). You may adjust the numbers per your need.
    Non-interactive Batch Job (Serial Run)

    batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

    Below is the example batch script (   job.txt ) for a serial run:

    # MAGMA Example Batch Script for the Basic Tutorial in the MAGMA manual
    #PBS -N 6pti
    #PBS -l nodes=1:ppn=28
    #PBS -l walltime=0:20:00
    module load cuda
    module load magma
    # Use TMPDIR for best performance.
    cd $TMPDIR
    # PBS_O_WORKDIR refers to the directory from which the job was submitted.
    cp $PBS_O_WORKDIR/example.c .
    icc $MAGMA_CFLAGS example.c
    

    In order to run it via the batch system, submit the  job.txt   file with the command:  qsub job.txt   

    Further Reading

    Tag: 
    Service: 
    Technologies: 
    Fields of Science: 

    MATLAB

    MATLAB is a technical computing environment for high-performance numeric computation and visualization. MATLAB integrates numerical analysis, matrix computation, signal processing, and graphics in an easy-to-use environment where problems and solutions are expressed just as they are written mathematically--without traditional programming.

    Availability and Restrictions

    MATLAB is available on Oakley, Ruby, and Owens Clusters. The versions currently available at OSC are:

    Version Oakley Ruby Owens notes
    2013a X     Default version on Oakley prior to 09/15/2015
    2013b X      
    2014a X X   Default version on Ruby prior to 09/15/2015
    2014b X X    
    2015a X X    
    2015b X X X  
    2016b X* X* X*  
    2017a X X X  
    *: Current default version

    You can use module spider matlab  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access to Academic Users

    All academic users must be added to the license server before using MATLAB.  Please contact OSC Help to be granted access.

    Another method to use MATLAB on OSC is to use the MATLAB Distributed Computing Server on Oakley via your local MATLAB client.  In order to use this feature, you should have your own MATLAB license, including the Parallel Computing Toolbox. See HOW-TO: Configure the MATLAB Parallel Computing Toolbox.

    Usage on Oakley

    Set-up

    To load the default version of MATLAB module, use  module load matlab . For a list of all available MATLAB versions and the format expected, type: module spider matlab . To select a particular software version, use   module load matlab/version . For example, use  module load matlab/r2014b to load MATLAB version 2014b. 

    Running MATLAB

    The following command will start an interactive, command line version of MATLAB:

    matlab -nodisplay 
    
    If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab . For more information about the matlab command usage, type matlab –h  for a complete list of command line options.

    The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

    Batch Usage

    When you log into oakley.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

    Interactive Batch Session
    For an interactive batch session using the command line version of MATLAB, one can run the following command:
    qsub -I -X -l nodes=1:ppn=12 -l walltime=00:20:00 
    

    which requests one whole node with 12 cores ( -l nodes=1:ppn=12 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above. The  –X flag enables X11 forwarding on the compute node, so you can use the MATLAB GUI if you choose.You may adjust the numbers per your need.

    Non-interactive Batch Job (Serial Run)

    In order to run MATLAB non-interactively via the batch system, you will require a batch submission script and a MATLAB M-file containing the script that will be run via the batch system. A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is an example batch submission script and a simple M-file. The batch script runs the M-file via the batch system.Example batch script (  job.txt  ) for a serial run:

    #PBS -N matlab_example
    #PBS -l walltime=00:10:00
    #PBS -l nodes=1:ppn=12
    #PBS -j oe
    module load matlab
    matlab -nodisplay -nodesktop < hello.m
    # end of example file
    

    Example M-file, hello.m:

    %Example M-file for Hello World
    disp 'Hello World' 
    exit   
    % end of example file
    
    In order to run hello.m via the batch system, submit the  job.txt file with the following command:
    qsub job.txt 
    

    This will run  hello.m via the batch system, and all output from the running of the script will be saved in the output.txt file.

    Parallel Processing in MATLAB

    MATLAB supports both implicit multiprocessing (multithreading) and explicit multiprocessing across multiple nodes.  

    Multithreading

    Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

    The system will use a number of threads equal to the number of cores you request.  Therefore, if you request nodes=1:ppn=4, your job will only spawn four threads. 

    Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may request nodes=1:ppn=1 and include the option  -singleCompThread when running MATLAB. An example is given below:

    #PBS -N disable_multithreading
    #PBS -l walltime=00:10:00
    #PBS -l nodes=1:ppn=1
    #PBS -j oe
    module load matlab
    matlab -singleCompThread -nodisplay -nodesktop < hello.m
    # end of example file
    
    Parallel computing across multiple nodes

    You can accomplish parallel processing using multiple nodes by using the Parallel Computing Toolbox in conjunction with the MATLAB Distributed Computing Server.  For more information about configuration and usage, see HOW-TO: Configure the MATLAB Parallel Computing Toolbox.  

    Usage on Ruby

    Set-up

    To load the default version of MATLAB module, use  module load matlab . For a list of all available MATLAB versions and the format expected, type: module spider matlab . To select a particular software version, use   module load matlab/version . For example, use  module load matlab/r2014b to load MATLAB version 2014b. 

    Running MATLAB

    The following command will start an interactive, command line version of MATLAB:

    matlab -nodisplay 
    
    If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab . For more information about the matlab command usage, type matlab –h for a complete list of command line options.

    The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

    Batch Usage

    When you log into ruby.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session using the command line version of MATLAB, one can run the following command:
    qsub -I -X -l nodes=1:ppn=20 -l walltime=00:20:00 
    

    which requests one whole node with 20 cores ( -l nodes=1:ppn=20 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above.  The  –X  flag enables X11 forwarding on the compute node, so you can use the MATLAB GUI if you choose.You may adjust the numbers per your need.

    Parallel Processing in MATLAB

    MATLAB supports both implicit multiprocessing (multithreading) and explicit multiprocessing across multiple nodes.  

    Multithreading

    Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

    The system will run as many threads as there are cores on the nodes requested.  For the normal nodes, this mean matlab will start 20 threads to run on the 20 cores of the node.  This is due to the whole node scheduling policy of Ruby (see Batch Specifics).

    Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may include the option "   -singleCompThread " when running MATLAB. An example is given below:

    #PBS -N disable_multithreading
    #PBS -l walltime=00:10:00
    #PBS -l nodes=1:ppn=20
    #PBS -j oe
    module load matlab
    matlab -singleCompThread -nodisplay -nodesktop < hello.m
    # end of example file
    
    Parallel computing across multiple nodes

    Parallel computing across multiple nodes is only available on Oakley.

    Usage on Owens

    Set-up

    To load the default version of MATLAB module, use  module load matlab . For a list of all available MATLAB versions and the format expected, type: module spider matlab . To select a particular software version, use   module load matlab/version . For example, use  module load matlab/r2015b to load MATLAB version r2015b. 

    Running MATLAB

    The following command will start an interactive, command line version of MATLAB:

    matlab -nodisplay 
    
    If you are able to use X-11 forwarding and have enabled it in your SSH client software preferences, you can run MATLAB using the GUI by typing the command  matlab . For more information about the matlab command usage, type matlab –h for a complete list of command line options.

    The commands listed above will run MATLAB on the login node you are connected to. As the login node is a shared resource, running scripts that require significant computational resources will impact the usability of the cluster for others. As such, you should not use interactive MATLAB sessions on the login node for any significant computation. If your MATLAB script requires significant time, CPU power, or memory, you should run your code via the batch system.

    Batch Usage

    When you log into owens.osc.edu you are actually logged into a Linux box referred to as the login node. To gain access to the multiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request multiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

    Interactive Batch Session
    For an interactive batch session using the command line version of MATLAB, one can run the following command:
    qsub -I -X -l nodes=1:ppn=28 -l walltime=00:20:00 
    

    which requests one whole node with 28 cores ( -l nodes=1:ppn=28 ), for a walltime of 20 minutes ( -l walltime=00:20:00 ). Here you can run MATLAB interactively by loading the MATLAB module and running MATLAB with the options of your choice as described above.  The  –X  flag enables X11 forwarding on the compute node, so you can use the MATLAB GUI if you choose.You may adjust the numbers per your need.

    Parallel Processing in MATLAB

    MATLAB supports both implicit multiprocessing (multithreading) and explicit multiprocessing across multiple nodes.  

    Multithreading

    Multithreading allows some functions in MATLAB to distribute the work load between cores of the node that your job is running on. By default, all of the current versions of MATLAB available on the OSC clusters have multithreading enabled. 

    The system will run as many threads as there are cores on the nodes requested.  For the normal nodes, this mean matlab will start 28 threads to run on the 28 cores of the node.  This is due to the whole node scheduling policy of Owens (see Batch Specifics).

    Multithreading increases the speed of some linear algebra routines, but if you would like to disable multithreading you may include the option "   -singleCompThread " when running MATLAB. An example is given below:

    #PBS -N disable_multithreading
    #PBS -l walltime=00:10:00
    #PBS -l nodes=1:ppn=28
    #PBS -j oe
    module load matlab
    matlab -singleCompThread -nodisplay -nodesktop < hello.m
    # end of example file
    
    Parallel computing across multiple nodes

    Parallel computing across multiple nodes is only available on Oakley.

    MATLAB with a GPU

    A GPU can be utilized for MATLAB. You can acquire a GPU by nodes=1:ppn=20;gpus=1 for Ruby or Owens (ppn=28 for Owens). For more detail, please read here.

    You can check the GPU assigned to you using:

    gpuDeviceCount  # show how many GPUs you have
    gpuDevice       # show the details of the GPU
    

    You can replace an array to gpuArray, for example:

    A = gpuArray(A)
    

    where A is a regular MATLAB array. This transfers the array data to the GPU memory. Then, you can use the gpuArray variable in GPU supported built-in functions. You can find the full list of GPU supported built-in functions from here. For more information about GPU programming for MATLAB, please read "GPU Computing" from Mathworks.

    Toolboxes and Features

    OSC's current licenses support the following MATLAB toolboxes and features (please contact OSC Help for license-specific questions):

    Polyspace Bug Finder
    PolySpace_Bug_Finder_Engine
    Aerospace_Blockset
    Aerospace_Toolbox
    Antenna_Toolbox
    Audio_System_Toolbox
    Automated_Driving_Toolbox
    Bioinformatics_Toolbox
    Communications System Toolbox
    Computer Vision System Toolbox
    Control System Toolbox
    Curve_Fitting_Toolbox
    DSP System Toolbox
    Data Acquisition Toolbox
    Database_Toolbox
    Datafeed_Toolbox
    Econometrics_Toolbox
    RTW_Embedded_Coder
    Filter_Design_HDL_Coder
    Financial Instruments Toolbox
    Financial_Toolbox
    Fixed_Point_Toolbox
    Fuzzy Logic Toolbox
    Global Optimization Toolbox
    Simulink_HDL_Coder
    EDA_Simulator_Link
    Image_Acquisition_Toolbox
    Image Processing Toolbox
    Instrument Control Toolbox
    LTE System Toolbox
    MATLAB_Coder
    MATLAB_Builder_for_Java
    Compiler
    MATLAB Report Generator
    Mapping Toolbox
    Model Predictive Control Toolbox
    Model-Based Calibration Toolbox
    Neural_Network_Toolbox
    OPC_Toolbox
    Optimization_Toolbox
    Distrib_Computing_Toolbox
    Partial Differential Equation Toolbox
    Phased_Array_System_Toolbox
    Polyspace Code Prover
    Powertrain_Blockset
    RF_Blockset
    RF_Toolbox
    Risk_Management_Toolbox
    Robotics_System_Toolbox
    Robust Control Toolbox
    Signal Processing Toolbox
    SimBiology
    SimEvents
    Simscape Driveline
    Simscape Electronics
    Simscape Fluids
    Simscape Multibody
    Simscape Power Systems
    Simscape
    Virtual_Reality_Toolbox
    Simulink_Code_Inspector
    Real-Time_Workshop
    Simulink_Control_Design
    Simulink Design Optimization
    Simulink_Design_Verifier
    Simulink Desktop Real-Time
    Simulink_PLC_Coder
    XPC_Target
    Simulink Report Generator
    Simulink_Test
    Simulink Verification and Validation
    Excel_Link
    Stateflow
    Statistics and Machine Learning Toolbox
    Symbolic Math Toolbox
    System Identification Toolbox
    Trading_Toolbox
    Vehicle_Network_Toolbox
    Vision_HDL_Toolbox
    WLAN_System_Toolbox
    Wavelet_Toolbox

    Further Reading

    Official PDF documentation can be obtained from the MathWorks Website

    Service: 
    Fields of Science: 

    MIRA

    MIRA - Sequence assembler and sequence mapping for whole genome shotgun and EST / RNASeq sequencing data. Can use Sanger, 454, Illumina and IonTorrent data. PacBio: CCS and error corrected data usable, uncorrected not yet.

    Availability and Restrictions

    The following versions of MIRA are available on OSC clusters:

    Version Oakley Owens
    4.0.2 X* X*
    *: Current default version

    You can use  module spider mira  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    MIRA is available to all OSC users without restriction.

    Usage on Oakley

    Set-up

    To configure your environment for use of MIRA, run the following command:   module load mira. The default version will be loaded. To select a particular MIRA version, use   module load mira/version. For example, use  module load mira/4.0.2 to load MIRA 4.0.2.

    Usage on Owens

    Set-up

    To configure your environment for use of MIRA, run the following command:   module load mira. The default version will be loaded. To select a particular MIRA version, use   module load mira/version. For example, use module load mira/4.0.2 to load MIRA 4.0.2.

    Further Reading

    Service: 
    Fields of Science: 

    MKL - Intel Math Kernel Library

    High-performance, multithreaded mathematics libraries for linear algebra, fast Fourier transforms, vector math, and more.

    Availability and Restrictions

    OSC supports single-process use of MKL for LAPACK and BLAS levels one through three. For multi-process applications, we also support the ScaLAPACK, FFTW2, and FFTW3 MKL wrappers. MKL modules are available for the Intel, GNU, and PGI compilers. MKL is available on Oakley, Ruby, and Owens Clusters. The versions currently available at OSC are:

    Version Oakley Ruby owens notes
    10.0.3        
    10.3.0 X     Default version on Oakley prior to 09/15/2015
    11.1.0 X      
    11.2.0   X   Default version on Ruby prior to 09/15/2015
    11.2.1   X    
    11.2.3 X* X*    
    11.3.2     X  
    11.3.3 X X X*  
    * : Current Default Version

    You can use module spider mkl to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

    Access

    MKL is available to all OSC users.

    Usage on Oakley

    Set-up

    To load the default MKL, run the following command: module load mkl. To load a particular version, use  module load mkl/version . For example, use  module load mkl/11.2.3  to load MKL version 11.2.3. You can use module spider mkl  to view available modules.

    This step is required for both building and running MKL applications.

    Exception: The "mkl" module is usually not needed when using the Intel compilers; just use the "-mkl" flag on the compile and link steps.

    Building With MKL

    The MKL module will automatically configure your environment to locate the appropriate include files. When linking, however, you have a variety of options to choose from depending on your needs. These options come in the form of environment variables defined by the module. Which variable you include when linking will determine what particular configuration of MKL you get.

    Environment Variable Function
    $MKL_CFLAGS Include flags for C
    $MKL_FFLAGS Include flags for Fortran
    $MKL_LIBS Use multithreaded MKL with 32-bit integers. This is the standard configuration.
    $MKL_LIBS_INT64 Use multithreaded MKL with 64-bit (“ILP64”) integers.
    $MKL_LIBS_SEQ Use single-threaded (“ SEQ uential”) MKL with with 32-bit integers.
    $MKL_LIBS_SEQ_INT64 Use single-threaded MKL with 64-bit integers.

    Notes:

    • 64-bit integers are not supported by the FFTW2 wrappers. (See below.)
    • The Intel compilers are specially configured for Intel MKL. If you are using this toolchain, you can use the “ -mkl ” option when linking in place of $MKL_LIBS .
    • The default, multithreaded MKL libraries will automatically run in a single thread if they are called from an OpenMP parallel region. If you want to force single-threaded behavior throughout your program, choose one of the “ _SEQ ” variables from the list above.
    Fortran 95 BLAS/LAPACK Wrappers

    To compile Fortran 95 programs with modules, add the $MKL_F95FLAGS variable to your compilation step. If you need 64-bit integers, use the $MKL_F95_FLAGS_INT64 instead. When linking, you will also need to use $MKL_F95LIBS (or $MKL_F95LIBS_INT64 if using 64-bit integers). These variables will allow your programs to use the BLAS and LAPACK wrappers for MKL.

    FFTW Wrappers

    A number of “wrappers” are provided in the form of environment variables that allow you to use FFTW APIs with MKL. Variables ending in FLAGS should be included with your compilation step, while variables ending in LIBS should be included in your linking step.

    Environment Variable Function
    $MKL_FFTW_CFLAGS Compile variable for C programs.
    $MKL_FFTW_FFLAGS Compile variable for Fortran programs.
    $MKL_FFTW2_D_CLIBS Linking variable for double-precision FFTW2 C programs.
    $MKL_FFTW2_D_FLIBS Linking variable for double-precision FFTW2 Fortran programs.
    $MKL_FFTW2_S_CLIBS Linking variable for single-precision FFTW2 C programs.
    $MKL_FFTW2_S_FLIBS Linking variable for single-precision FFTW2 Fortran programs.
    $MKL_FFTW3_CLIBS Linking variable for FFTW3 C programs.
    $MKL_FFTW3_FLIBS Linking variable for FFTW3 Fortran programs.

    Notes:

    • FFTW3 is the default wrapper and should be picked up by your linker automatically. If you would prefer to use FFTW2, use the appropriate linking variable.
    • The FFTW2 wrappers do not support 64-bit (“ILP64”) integers.
    • <