Once your account is set-up, you will receive a welcome letter from us containing your username and instructions for setting up your account.

MyOSC, one of our web portals, provides services to OSC users, including:

• Password changes and resets for your HPC accounts
• Changing your UNIX shell
• Updating your email address
• Tracking your project(s) balance​

Note: HPC password changes and resets take 3-5 minutes to process. If you attempt to login with your new password, you will receive an error. Too many failed login attempts results in your account being automatically locked for 5 minutes.

Are the answers you're looking for not here? Contact OSC Help. 

Principal Investigators of OSC projects are responsible for updating their authorized user list, their outside funding sources, and their publications and presentations that cite OSC. All of these tasks can be accomplished by contacting OSC Help with the information. PIs are also responsible for monitoring their project's Resource Unit balance and for submitting a proposal to the Allocations Committee of the Statewide Users Group before the balance goes negative. Projects with negative balances are subject to restrictions, causing submitted jobs to have lower priority in the queue. We have a resource page for your convenience. 

Allocation managment for Ohio-based academic projects

Eligible Principal Investigators must be tenure-track faculty members or permanent research scientists at Ohio colleges or universities. All other researchers, such as postdoctoral researchers, graduate students, and undergraduate students, may be authorized users under a project led by an eligible PI.

Principal Investigators who plan to submit a proposal for additional resource units should have the following information ready when they fill out the application form:

• names, departments, affiliations, and email addresses of new authorized users
• names and email address of two or three suggested peer reviewers
• document with performance reports and justification for request
• electronic version of resume/CV
• list of publications and presentations that cite OSC
• proposal text
• document with other pertinent information (optional)

Send a query to grant@osc.edu or to oschelp@osc.edu if you require more detailed information concerning the application process.

Once a proposal has been approved for peer review, any restrictions that have been accrued due to a negative balance will be lifted. The Allocations Committee meets every other month (even-numbered months), and awards are made after the meeting.

Account Maintenance and Usage: Please use MyOSC to update your password, shell, and check usage on the project. The command OSCusage will also provide usage details. 

Allocation managment for purchased cycles

Please disregard any RU messages or instructions to apply for additional resources. If you have any billing questions, please consult your contact at OSC or inquire with OSC Help.

Applying for Academic Projects

Statewide academic allocation of computing resources and services is a peer-reviewed account process governed by the Allocations Committee of the Statewide Users Group and subject to the Ohio Supercomputer Center (OSC) Code of Ethics for Academic Users.

The procedures described enable services to be available to approved projects conducted by eligible principal investigators (PIs) at Ohio academic institutions. In order to be an eligible PI at OSC, you must be eligible to hold PI status at your college, university, or research organization administered by an Ohio academic institution (i.e., be a full-time, permanent academic researcher or tenure-track faculty member at an Ohio college or unviersity). Students, post-doctoral fellows, visiting scientists, and others who wish to use the facilities may be authorized users on projects headed by an eligible PI. Once a PI has received their project information, he/she can contact OSC Help to add new authorized user(s) to a project at any time.

All first-time PIs must submit a signature page (electronically or hard copy). Please send a copy of the signature page to the attention of the Allocations Committee, OSC, 1224 Kinnear Road, Columbus, Ohio, 43212.

OSC's online project application is part of an electronic system that leads you through the process step by step. Before you begin to fill in the application form, especially if you are new to the process, look at the Project Application Help page. From this help page, please note the required materials for each request. Have all information and materials (suggested reviewer names and addresses, proposal text file names, etc.) at hand before you begin the application form. You can save a partially completed project application form for later use. As a reminder, you cannot re-submit application materials if they have already been used. 

Online Project and Account Application Form (NEW OSC PI)

Online Project Application Form (EXISTING PI)

Classroom Project Application Form (use our resources and software in the classroom)

If you need assistance, please contact OSC Help.

Applying for Commercial Access

Researchers from commercial organizations who wish to use the OSC's resources should complete the Ohio business request form or request access through the form for someone other than an Ohio business or academic researcher.

Add a new user

Please submit your request to OSC Help or use our online submission form. For reference, we will need the following:

Project:
Full Name:
If not U.S. Citizen, country of citizenship:
Institution:
Department:
Affiliation:
Email address:
Special Requests?

Letter of Commitment for Outside Funding Proposals

OSC will provide a letter of commitment users can include with their account proposals for outside funding, such as from the Department of Energy, National Institutes of Health, National Science Foundation (limited to standard text, per NSF policy), etc. This letter details OSC's commitment to supporting research efforts of its users and the facilities and platforms we provide our users. [Note: This letter does not waive the normal OSC account application process; it merely states that OSC is willing to support such application.] The information users must provide for the letter is:

• address to which the letter should be directed (e.g. NSF, Department, mailing address)
• name of funding agency's representative
• name of the proposal, as well as names and home institutions of the Co-PIs
• number of RUs per year you would apply for if you were to receive funding
• number of years of proposed research
• solicitation number

Send e-mail with your request for the commitment letter to OSC Help or submit online. We will prepare a draft for your approval and then we will send you the final PDF for your proposal submission. Please allow at least two working days for this service.

Letter of Support for Outside Funding Proposals

Letters of support may be subject to strict and specific guidelines, and may not be accepted by your funding agency.

If you need a letter of support, please see above "Letter of Commitment for Outside Funding Proposals". 

Applying at NSF Centers

Researchers requiring additional computing resources should consider applying for allocations at National Science Foundation Centers. For more information, please write to oschelp@osc.edu, and your inquiry will be directed to the appropriate staff member.:

We require you cite OSC in any publications or reports that result from projects supported by our services.

While our python installations come with many popular packages installed, you may come upon a case where you need an addiditonal package that is not installed.  If the specific package you are looking for is available from Anaconda.org (formerlly binstar.org) you can easily install it and required dependencies by using the Conda package manager.

To be able to install a package using the conda package manager:

• You must use a Anaconda distribution of python:
  • On Oakley the following modules:
    • python/2.7.latest*, python/2.7.8, python/3.4.2
  • On Ruby:
    • python/2.7.latest*, python/2.7.8, python/3.4.2
  • On Owens:
    • python/2.7, python/3.5
• * = latest suffix refers to a distribution that will be electively updated
• Package should be available through Anaconda.org and/or using pip (see next section).

If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

But I use virtualenv and/or pip!

See the comparison to these package management tools here:

http://conda.pydata.org/docs/_downloads/conda-pip-virtualenv-translator.html

Pip installations are supported:

module load python
module list # check which python you just loaded
pip install --upgrade --user PACKAGE # where PACKAGE is a valid package name

If binaries exist in a pip installation they are usually installed in:

$HOME/.local/bin

The one issue with this approach is portability with multiple Python modules.  If you plan to stick with a single Python module, then this should not be an issue.  However, if you commonly switch between Python 2 and Python 3, then be aware of the potential trouble in using the same installation location for all Python versions.

Virtualenv may also work, but is not under active testing at OSC.

Procedure

We will install the yt package to a local directory in this example.

Load proper python module

module load python/2.7.8

Clone python installation to local directory

conda create -n local --clone="$PYTHON_HOME"

This will clone the entire python installation to ~/envs/local directory. The process will take serveral minutes.

conda create -n local

This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.

conda create -n local python={version} anaconda

If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=2.4" for version 2.4.

Activate clone environment

source activate local

On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

conda remove conda-build
conda remove conda-env

Install package

conda install yt

• Replace yt with the name of the package you want to install, as listed by anaconda.org.

Test python package

Now we will test our installed python package by loading it in python and checking its location to ensure we are using the correct version.

python -c "import yt;print yt.__file__"

Output:

$HOME/.conda/envs/local/lib/python2.7/site-packages/yt/__init__.py

• Replace both instances of yt  with the name of the package you installed.

Install your own python modules

If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

Further Reading:

Conda Test Drive: https://conda.io/docs/test-drive.html 

Introduction

The MATLAB Parallel Computing Toolbox and Distributed Computing Server are designed to allow users to create and launch parallel MATLAB jobs on a cluster of compute nodes. It also allows users to remotely connect to OSC resources, whether to run parallel jobs in MATLAB or to use toolboxes for which users own their own licenses. This guide will explain the basics of how to configure your Parallel Computing Toolbox for OSC systems.

Versions

The following versions of the MATLAB Parallel Computing Toolbox are supported at OSC:

Usage Overview

When you use the MATLAB Parallel Computing Toolbox, you have a MATLAB client session and one or more MATLAB workers. The client session may run on your laptop/desktop computer ("remote client") or it may run on an OSC login, OnDemand, or compute node. The MATLAB workers always run on the OSC cluster as part of a batch job. In the client session you will run MATLAB commands to set up and run a batch job, and MATLAB submits the job for you.

This document describes how to perform a computation on a single worker (one processor) or on multiple workers using a script with a "parfor" loop.

Licensing issues

You must have a license for MATLAB and the MATLAB PCT to allow you to run the MATLAB client session on your laptop/desktop computer. OSC has licenses for the MATLAB Distributed Computing Server, which covers the MATLAB workers running on the cluster.

OSC is included in the OSU site license for MATLAB, so OSC users have the option of running the client session on an OSC machine. You can manage and view your MATLAB licenses using the Mathworks License Center.

In any case, in order to use MDCS license on OSC, you need to be added to the license server. So, please contact OSC Help

Remote client

If you run your MATLAB client session on your local computer, it is considered a remote client. You will be able to use any toolboxes that you have a license for by submitting batch jobs from your MATLAB client. The batch jobs you submit from your MATLAB client through the PCT will be able to utilize functions such as "parfor" and "spmd" to run your code in parallel, in addition to being able to run normal MATLAB code.

Client running on Oakley

Through the MATLAB installations on Oakley, you will be able to take advantage of the interactive capabilities in the MATLAB PCT, as well as submitting batch jobs through the MATLAB client sessions. OnDemand or VNC is needed to use this (the MATLAB PCT is GUI only at this time), but it will allow you to debug your parallel jobs using various tools in real time. Jobs submitted through the client sessions on Oakley behave the same as jobs submitted by qsub, but can utilize the PCT functions such as "parfor" and "spmd".

Performance limitations

Parallel MATLAB on a cluster has a lot of overhead. It may not give you the speedup you expect, especially if you're solving small problems. Another consideration is that the MATLAB workers are single-threaded. They don't take advantage of the multithreading built into many MATLAB functions.

Download and Install the Configuration Files

The first step is to download the necessary configuration files to the computer where you will be running the MATLAB client session.

Click the link below to download the files.

OSCMatlabPCT

OSCMatlabPCT Configuration Resources -- Directory Structure

OSCMatlabPCT (top-level directory)

• config - A directory containing the cluster profiles and configuration functions for using the MATLAB PCT at  OSC.  Note: the only function that users should edit is the "addSubmitArgs" function, as this allows setting some additional batch options, such as walltime, email, and custom job name.  All the other functions have been specially prepared by OSC staff to allow MATLAB to work in harmony with the HPC system on Oakley.
• launch - A directory containing two scripts, "client_session_script.m" for launching jobs and "reconnect_client_session.m" for reconnecting to a job.  These scripts illustrate key concepts of using the PCT when accessing our system remotely via the MATLAB client on your personal computer.  They also apply to submitting batch jobs from a MATLAB client running in OnDemand.  Both scripts are heavily commented and give usage details beyond what this document can provide.  Note that these scripts contain commands to be run in your client session; do not submit them using the batch command.
• PCTtestfiles -  A directory containing parfor use-cases. The first, "eigtest", is an example of how to program an extremely simple entry function using parfor.  It simply computes the eigenvalues of multiple large, random matrices.  The second case, is an example of how to run a parallel Simulink simulation using parfor.  The entry function is "paralleltestv2", which calls the function "parsim" inside a parfor loop.  The "parsim" script contains commands that initialize Simulink and run the simulation on each of the parfor workers.

Import and Configure the Cluster Profile

These configuration steps need to be done once before you use PCT.  (If you upgrade to a new version of MATLAB you'll have to reconfigure.)  There are also command line options for performing these tasks.

1. After launching MATLAB, click the "Parallel" dropdown menu from the "Environment" menu and select "Manage Cluster Profiles".  At this time, a new window should open displaying the Cluster Profile Manager.  

2. In the Cluster Profile Manager window, click the "Import" button and locate the "clusterProfiles" directory contained within the OSCMatlabPCT/config/clusterProfiles directory.  

Cluster profiles are named according to filesystem configuration and version.  Select the file which corresponds to your current version of MATLAB and your filesystem configuration.  If you're running remotely on your laptop/desktop computer select a "NonShared" version.  If you're running on Oakley, select a "Shared" version.  Click "Open" to proceed.  

3. If you are using a shared filesystem configuration: No further modifications will need to be made to the cluster profile, and you can close the Cluster Profile Manager and continue with configuring your job. 

If you are using a non-shared filesystem configuration: You will need to make some changes to your cluster profile before exiting the Cluster Profile Manager.  Click the "Edit" button to enable editing.

4. In the editing window under "Submit Functions", you should see two entries -- IndependentSubmitFcn and CommunicatingSubmitFcnIntel.  These fields contain cell arrays with three values: a handle to the submit function, the hostname of the cluster, and the remote job storage location where results files and logs will be stored.  You only need to change the remote job storage location.  Use an absolute path to a location in your OSC home directory where you'd like the output of your job (or jobs) to be retained.  

Here is an example of the full syntax:

{@independentSubmitFcn, 'oakley.osc.edu', '/my/home/dir/MATLAB'}

{@communicatingSubmitFcnIntel, 'oakley.osc.edu', '/my/home/dir/MATLAB'}

Note:  If you try to validate your configuration, the last test will always fail for remote clients, even if the configuration is correct.

5. Include the configuration files in your MATLAB path.  Click on "Set Path", then "Add Folder".  Locate and select the OSCMatlabPCT\config directory.  Click "Select Folder" and "Save".  (Alternatively you can use the "addpath" command in the MATLAB command window.)

Configure and Run a Batch Job

The primary way to utilize the MATLAB PCT at OSC is to submit batch jobs from the MATLAB client, either on your local PC or on Oakley via OnDemand.  These instructions assume that you have already configured your MATLAB client as described above.

1. Write your script to be run on the cluster.  This can be just a serial script or it may contain parallel commands such as "parfor".  Some example scripts are provided in the directory OSCMatlabPCT/PCTtestfiles.  The example below uses "eigtest.m" as the script to be run.
2. In your client session:  Using "client_session_script.m" as a guide, run the commands to connect to the cluster and launch a batch job.  We don't recommend that you run the script as written, at least at first.  Until you're familiar with using the PCT, simply copy commands out of the script and paste them into your MATLAB command window as needed.  There are two important functions involved in launching a job.  The command "parcluster" creates a cluster object in the MATLAB workspace, which is used to store information sent between the remote cluster and your MATLAB client.  The command "batch" begins an automated process that can connect to the Oakley cluster, submit a job to PBS, and initialize the MATLAB Distributed Computing Server (MDCS).  For more on the batch command, see the "client_session_script.m" file, and the MATLAB help.
3. When your job has been successfully configured, you will be prompted for your username and password at OSC.  After entering this, your job will be submitted to the system, and assigned an OSC job ID.  Please note: as explained in the "client_session_script" example, MATLAB remembers your jobs on the cluster by the job index number, not the OSC job ID.  Also, if you are unable to qstat your job immediately after it is submitted, don't worry!  There seems to be some latency between the MATLAB MDCS and the batch system when it comes to exchanging job status info, but this should not affect your ability to track your job.  In fact, as shown in the example, there are several ways to get information about your jobs directly from your MATLAB client. After submission you can check your job's progress or retrieve results at any time (even after closing the MATLAB client), and you can continue using the MATLAB client for other work if you wish.
4. Wait for your job to complete.  You may use the "wait" command in your client session or simply monitor your job's status with "qstat" on an Oakley login node to know when your job has completed.
5. Reconnect after closing your MATLAB client session.  The script "reconnect_client_session.m" illustrates how to find your job information again if you restart your MATLAB client session.
6. Retrieve your results.  The "load" or "fetchOutputs" command retrieves the results of your computations from your MATLAB working directory on Oakley after the job has finished.  They also report errors if your run failed.  "Load" is used for exporting the workspace from an entry script, and "fetchOutputs" is used for retrieving the specified output arguments of an entry function.  For large data, there is an important caveat: MATLAB does not utilize sftp when fetching your files, so large files might take a long time to retrieve.  Additionally, if your output arguments/workspace is over a certain size (around 2GB), using the "load" or "fetchOutputs" commands will give an "index out of range" error caused by the MDCS failing to save the output to file.  This is due to an internal limitation in the default version of MATLAB's ".mat" files.  However, you may work around this limitation by manually saving your workspace or arguments in your entry script, using the '-v7.3' switch in the "save" command.

Specifying Additional PBS Options for Your Batch Job

The submit functions are MATLAB functions which serve two primary purposes: to create the appropriate connections to the cluster where your job will run, and to procedurally generate and submit a shell script to the queue on Oakley.  Your nodes and cores (ppn) are determined by the number of MATLAB workers you request in your submit script, however to change or add additional arguments or resource requests you must add the PBS commands to the string in the function "addSubmitArgs" (located in config).  The default arguments in the string are for a walltime of 1:00:00, a job name "default", and emails sent on abort, begin, and end of the job.  This function is called in the submit functions, so advanced users may choose to do what they wish with how this file is called in the submit functions to streamline their workflow if multiple configurations are needed.

Configure and Run an Interactive Job

There are a couple useful ways for OSU users to use the interactive features of the MATLAB PCT.  Because of our license terms, this option is available only to OSU faculty, staff, and students.

Using OnDemand with a Shared Cluster Profile

OSU users may take advantage of the MATLAB licenses on Oakley to submit interactive "parpool" (R2013b or later) or "matlabpool" (R2013a) jobs the batch system.  These MATLAB PCT functions allow you to run parallel functions such as "parfor" or "spmd" directly from command line or a script, and allow users to debug or interact with parallel jobs in real time.  OSC offers this functionality in MATLAB clients running through the OnDemand portal.  In this method, simply start on Oakley desktop session in OnDemand, open a terminal window, load the correct module, and start MATLAB.  Be sure the OSCMatlabPCT/config directory is in your path.  You will then import the correct shared cluster profile for your version of MATLAB, and use the "parpool" (R2013b or later) or "matlabpool" (R2013a) command to initialize an automatic parallel pool. This might take a while to set up, as the pool request must go throught the batch system like everything else, so be patient!  After a couple minutes or more (depending on the resource request), your pool will connect to the MATLAB workers, and you may begin running parallel commands. This will allow simulation of how jobs will behave when submitted to the MDCS as a batch job.

Using VNC with the "local" Cluster Profile

Another way to run an interactive parallel job using the MATLAB PCT is to submit an Interactive Batch job from using qsub, and run MATLAB on a compute node using VNC to utilize the GUI.  You can then import the "local" cluster profile, which runs the workers directly on the node.  Only one node can be used, but workers will run on multiple processors on the node.  This can be used in situations when using OnDemand is not ideal, but users should carefully follow the directions provided so as to not leave VNC processes behind on the node.  Directions for using VNC in a batch job are located here: https://osc.edu/documentation/howto/use-vnc-in-a-batch-job">https://osc.edu/documentation/howto/use-vnc-in-a-batch-job.

Additional Documentation

www.mathworks.com/index.html?s_tid=gn_logo">http://www.mathworks.com/index.html?s_tid=gn_logo">Mathworks has written extensive documentation covering all of the public functions within the Parallel Computing Toolbox. The PCT homepage is located at www.mathworks.com/products/parallel-computing/">http://www.mathworks.com/products/parallel-computing/">The Mathworks Parallel Computing Toolbox Homepage.

For more information about how to construct and run an independent job, see the Mathworks documentation page "www.mathworks.com/help/distcomp/program-independent-jobs.html">http://www.mathworks.com/help/distcomp/program-independent-jobs.html">Program Independent Jobs".  

For more information about how to construct and run a communicating job, see the Mathworks documentation page "www.mathworks.com/help/distcomp/introduction.html">http://www.mathworks.com/help/distcomp/introduction.html">Program Communicating Jobs".

Errors

If you have any errors related to the OSC MATLAB PCT configuration files, or have any other questions about utilizing the MATLAB PCT at OSC please contact OSC Help with: your user id, any relevant error messages, a job ID(s) if applicable, and the version of MATLAB you are using.

To connect to OSC services, a secure tunnel to a session is required. This can be done relatively simply in OSX and Linux by using the SSH functionality built into the system, but Windows users have had to configure and use third party applications like PuTTY or Java to access secure resources at OSC. OSC Connect is a native windows application written in C# and compiled for .NET 2.0, providing preconfigured management of secure tunnel connections for Windows users, as well as providing a launcher for secure file transfer, VNC, terminal, and web based services. 

OSC Connect is supported on Windows versions from Windows XP through Windows 10.

Getting Started

1. Download the latest release of OSC Connect from here: Latest Build
   • Click OSCConnect.exe  and download. Use "Save link as" to download this file to a folder of your choice. 
   • See here for all the previous releases. 
2. Double-Click the OSCConnect.exe icon to run the application. In the current state, OSC Connect is entirely deployed by a single executable file; no further installation is required.   ​
3. When you run OSC Connect, a temporary folder named "ConnectFiles" with four additional files will be created. These are required for proper functionality of the application. Please ensure that these files are permitted by your IT administrator:
   • plink.exe is the command-line version of PuTTY used by the application to create the secure connection to OSC resources.
   • putty.exe  is the GUI application of PuTTY used to provide terminal emulation remote console connections to OSC resources.
   • vncviewer.exe is the VNC viewer client used to view a remote desktop session.
   • WinSCP.exe  is the SFTP client used for file transfer.

4. Once your connections to OSC services as well as the OSC Connect app is closed, the temporary folder named "ConnectFiles" will be removed automatically. 

OSC Connect Graphical User Interface

After you double-click the  OSCConnect.exe  icon  , the application graphical user interface is shown as below:

• Network Status: it indicates which OSC cluster you will be connected to. The option can be changed in "Settings". 

• Settings: it provides several configuration options to modify the behavior of the application. 

  • Connection Settings: use this dropdown to select the default host/cluster. Selecting a server here will change the endpoint for tunneling, sftp connections, console connections, and connectivity checking.

  • System Settings: 

    • Detect Clipboard Activity: when this option is enabled, the application will detect valid data on the Windows clipboard and populate the application. ( Default: Off )

    • Check for New Versions: when this option is enabled, the application will check for version updates. (Default: on)

  • Automation Settings:

    • Save User Credentials: when this option is enabled, it allows the application to remember the user when the application is reopened. This saves the user credentials to the user settings using DPAPI Encryption. Passwords are decrypted only by current the Windows user account. ( Default: Off )

    • Launch Tunnel On Import: when this option is enabled, the tunnel will automatically connect when the application detects a valid clipboard string and the user credentials have been entered. ( Default: On )

  • VNC Settings

• ​About: it provides additional information about OSC Connect.

After you provide your OSC Credentials, i.e. your OSC HPC username and password, more functionalities are available as shown below:

• SFTP: it uses WinSCP as the SFTP client to securely connect to the OSC file system over SFTP.
• Console: it uses PuTTY to provides you shell access to OSC cluster.

In addition, Session Type is provided such that advanced users can connect to a running session mannually.

Connecting to a Session

The OSC Connect application can be used to connect to a running session launched through OSC OnDemand.

1. Navigate to OSC Demand to create a new session under either "Desktops" or "Desktop Apps". 
   • More information on how to use OSC OnDemand can be found here
2. Once your session is running, you have two options to connect to the session: "noVNC Instructions" and "Native Client Instructions". Click "Native Client Instructions" to use OSC Connect. 
3. Follow the steps below to connect to the running session:
   • Step 1: Download the latest OSC Connect if you don't already have it. See "Getting Started" for more details. 
   • Step 2: Launch OSC Connect. 
   • Step 3: Click - osc://xxxxx. osc://xxxxx is a custom URI scheme that is registered when you launch the application. Simply click the link to populate the configuration information and connect to your running session. If OSCConnect.exe is not running when you click the URI, the OSC Connect application will be popped up. Enter your OSC HPC username and password and you will be able to connect to the session by clicking the "Connect" button.
   • For advanced users, you can skip step 3 and connect to a running session mannually. Depending on the type of application, you can choose either "iHPC VNC Desktop" to create a VNC tunnel or "Web Browser Session" to create a browser tunnel. 

FAQ

I've clicked the osc:// link and nothing happened.

Be sure to run OSCConnect.exe at least once. The initial launch will add a key to your user registry that initializes the URI scheme. If you move or rename the OSCConnect.exe file, you will need to run the application again manually to update the path in the handler.

I've received the error "Unable to open helper application. The protocol specified in this address is not valid."

This issue appears in some earlier versions of Internet Explorer when attempting to launch the application from a Temporary location. Download and run the OSCConnect.exe application, being sure to save the file to a non-temporary location.

An eligible principal investigator (PI) heads a project account and can authorize/remove user accounts under the project account (please check our Allocations and Accounts documentation for more details). This document shows you how to identify users on a project account and check the status of each user. 

Identify Users on a Project Account

If you know the project acccount

If you know the project account (projectID), the following command will list all users on the project:

getent group projectID

The returned information is in the format of:

projectID:*:gid: list of user IDs

gid is the group identifier number unique for the project account projectID. 

For example, the command  getent group PZS0530  lists all users on the project account PZS0530 as below:

bash-4.1$ getent group PZS0530
PZS0530:*:2959:yli,osc0539,elton,ananth,osc0413,dhudak,osc0695,ksaantha,changlee,buss,osc0414,osc0478,nsharma,kmanalo,nwadih,bsmith,...

If you don't know the project acccount, but know the user account

If you don't know the project account, but know the user account, userID, use the  groups  command to list all of the groups the userID belongs to:

groups userID

The returned information is in the format of:

userID : list of groups

where the first item after " userID : " is the primary group, i.e. the project account (projectID) which userID is on. Once you know the project account, use the command  getent group projectID  as discussed in the previous session to list all users on this project.

For example, with the userID as xwang, the command groups xwang  returns the information as below:

bash-4.1$ groups xwang
xwang : oscgen lstc gaussian ...

It lists all groups xwang belongs to, and oscgen is the project account which xwang is on. The command  getent group oscgen  lists all user accounts on the project account oscgen:

bash-4.1$ getent group oscgen
oscgen:*:200:jordan,jpu,mlewis,rmonroe,rmarshal,spears,sengupta,jtm,adraghi,lepage,ian,karin,remote,njustice,bedford,hjiang,tom,mudronja,elainep,gisadmin,airani,guilfoos,osc0498,osc0722,ngagnet,mfaerman,justinw,arya,mattm,echong,rahmed,jwright,...

If you don't know either the project acccount or user account

If you don't know either the project account or user account, you can use ldapsearch  command to get the user account based on the user's registration information such as name, email address, etc. 

Use the following command to list all of the user accounts accociated with the name First Last:

ldapsearch -x -LLL "(gecos=First Last)" | grep cn | awk '{print $2}'

Use the following command to list all of the user accounts accociated with the email address email@address:

ldapsearch -x -LLL "(mail=email@address)" | grep cn | awk '{print $2}'

For example, with user's first name as Summer and last name as Wang, the command

ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}' returns the information as below:

bash-4.1$ ldapsearch -x -LLL "(gecos=Summer Wang)" | grep cn | awk '{print $2}'
xwang
xwangnd
... 

With user's email address as xwang@osc.edu, the command ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}' returns the information as below:

bash-4.1$ ldapsearch -x -LLL "(mail=xwang@osc.edu)" | grep cn | awk '{print $2}'
xwang
xwangnd
...

Once you know the user account userID, follow the discussions in previous session to get all user accounts on the project. Please contact OSC Help if you have any questions. 

Check the Status of a User

Use the  finger  command to check the status of  a user account userID as below:

finger userID

For example, if the userID is xwang, the command  finger xwang  will return:

bash-4.1$ finger xwang
Login: xwang                            Name: Summer Wang
Directory: /nfs/17/xwang                Shell: /bin/bash
On since Tue Apr 19 09:58 (EDT) on pts/190 from localhost:43.0
Mail forwarded to xwang@osc.edu
No mail.
No Plan.

• The home directory of xwang is on /nfs/17 ( Directory: /nfs/17/xwang )
• The shell of xwang is bash ( Shell: /bin/bash ). If the information is Shell:/access/denied , it means this user account has been either archived or resticted. Please contact OSC Help if you'd like to reactivate this user account.
• xwang@osc.edu is the associated email with the user account xwang; that is, all OSC emails related to the account xwang will be sent to xwang@osc.edu ( Mail forwarded to xwang@osc.edu ). Please contact OSC Help if the email address asscoiated with this user account has been changed to ensure important notifications/messages/remindars from OSC may be received in a timely manner.

Check the Usage and Quota of a User's Home Directory/Project's Project Space

All users see their file system usage staticstics when logging in, like so:

As of 2016 Apr 20 04:02 userid userID on /nfs/nn used XGB of quota 500GB and Y files of quota 1000000 files

The information is from the file /usr/local/quotas/quota_report.txt, which is updated daily. Some users may see multiple lines asscoiated with userid, as well as information on project space usage and quota, if there is one. The usage and quota of the home diretory of userID is provided by the line including the file server your home directory is on (for more information, please visit Home Directories), while others (generated due to file copy) can be safely ingored. 

You can check any user's home directory or a project's project space usage and quota by running:

grep <userID OR projectID > /usr/local/quotas/quota_report.txt

This document shows you the steps to install R packages locally without root access on OSC's Oakley cluster. 

R comes with a single library  $R_HOME/library which contains the standard and recommended packages. This is usually in a system location. On Oakley cluster, it is   /usr/local/R/3.0.1/lib64/R/library . R also has a default value for a directory where users can install their own R packages. On Oakley cluster, it is ~/R/x86_64-unknown-linux-gnu-library/3.0 if the default R-3.0.1 module is loaded. This directory doesn't exist by default. The first time a user installs an R package, R will ask the user if s/he wants to use the default location and if yes, will create the directory.

A Simple Example

First you need to load the module for R:

module load R

On Oakley, the default R module is version 3.0.1 .

Then fire up an R session:

R

To install package lattice, use this command inside R:

> install.packages("lattice", repos="http://cran.r-project.org")

It gives a warning: 

Warning in install.packages("lattice") :
'lib = "/usr/local/R/3.0.1/lib64/R/library"' is not writable
Would you like to create a personal library
~/R/x86_64-unknown-linux-gnu-library/3.0
to install packages into?  (y/n) 

Answer y , and it will create the directory and install the package there.

Setting the Local R Library Path

If you want to use another location rather than the default location, for example, ~/local/R_libs/ ,  you need to create the directory first:

mkdir ~/local/R_libs

Then type the following command inside R:

> install.packages("lattice", repos="http://cran.r-project.org", lib="~/local/R_libs/")

It is a bit of burden having to type the long string of library path every time. To avoid doing that, you can create a file .Renviron in your home directory, and add the following line to the file:

export R_LIBS=~/local/R_libs/ 

Whenever R is started, the directory ~/local/R_libs/ is added to the list of places to look for R packages and so:

> install.packages("lattice", repos="http://cran.r-project.org")

will have the same effect as the previous install.packages() command. 

To see the directories where R searches for libraries, use the command:

>.libPaths();

Setting The Repository

When you install an R package, you are asked which repository R should use. To set the repository and avoid having to specify this at every package install, create a file .Rprofile in your home directory. This is the start up code for R. Add the following line to the file:

cat(".Rprofile: Setting R repository:")
repo = getOption("repos") 
# set up the server from which you will download the package.
repo["CRAN"] = "http://cran.case.edu" 
options(repos = repo)
rm(repo)

Now you only need to do 

> install.packages("lattice")

That will download the package lattice from  http://cran.case.edu and install it in ~/local/R_libs  .

Updating Packages

> update.packages() inside an R session is the simplest way to ensure that all the packages in your local R library are up to date. It downloads the list of available packages and their current versions, compares it with those installed and offers to fetch and install any that have later versions on the repositories.

Removing packages

> remove.packages("lattice") inside an R session to remove package lattice . An even easier way is just to go into the directory ~/local/R_libs and remove the directory lattice from there.

References

Add-on packages in R installation guide (http://cran.r-project.org/doc/manuals/R-admin.pdf)

While we provide a number of Perl modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Perl modules. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

CPAN Minus

CPAN, the Comprehensive Perl Achive Network, is the primary source for publishing and fetching the latest modules and libraries for the Perl programming language. The default method for installing Perl modules using the "CPAN Shell", provides users with a great deal of power and flexibility but at the cost of a complex configuration and inelegant default setup.

Setting Up CPAN Minus

To use CPAN Minus, we must first load it, if it hasn't already been loaded.

module load cpanminus

Next, in order to use cpanminus, you will need to run the following command only ONCE:

perl -I /usr/local/cpanminus/perl5/lib/perl5 -Mlocal::lib

Using CPAN Minus

In most cases, using CPAN Minus to install modules is as simple as issuing a command in the following form:

cpanm [Module::Name]

For example, below are three examples of installing perl modules:

cpanm Math::CDF
cpanm SET::IntervalTree
cpanm DB_File

Testing Perl Modules

To test a perl module import, here are some examples below:

perl -e "require Math::CDF"
perl -e "require Set::IntervallTree"
perl -e "require DB_File"

The modules are installed correctly if no output is printed.

What Local Modules are Installed in my Account?

To show the local modules you have installed in your user account:

perldoc perllocal

Reseting Module Collection

If you should ever want to start over with your perl module collection, delete the following folders:

rm -r ~/perl5 
rm -r ~/.cpanm

Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space.

Getting Started

Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:

    local
    |-- src
    |-- share
        `-- lmodfiles

This structure is how OSC organizes the software we provide. Each directory serves a specific purpose:

• local - Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
• local/src - Stores the installers -- generally source directories -- for your software. Also, stores the compressed archives ("tarballs") of your installers; useful if you want to reinstall later using different build options.
• local/share/lmodfiles - The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.

You can create this structure with one command.

NOTE: Ensure $HOME is the full path of your home directory. You can identify this from the command line with the command echo $HOME.

After navigating to where you want to create the directory structure, run:

    mkdir -p $HOME/local/src $HOME/local/share/lmodfiles

Installing Software

Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/local/src:

    cd $HOME/local/src
    wget https://github.com/git/git/archive/v2.9.0.tar.gz

Now extract the tar file:

    tar zxvf v2.9.0.tar.gz

Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/local/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

Then, we can build Git using the following commands:

    cd git-2.9.0
    autoconf # this creates the configure file
    ./configure --prefix=$HOME/local/git/2.9.0 CC=icc CFLAGS=-static-intel
    make && make install

Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

Creating a Module

Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

We will be useing the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

-- Local Variables
local name = "git"
local version = "2.9.0"

-- Locate Home Directory
local homedir = os.getenv("HOME")
local root = pathJoin(homedir, "local", name, version)

-- Set Basic Paths
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("MANPATH", pathJoin(root, "share/man"))

NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

Initializing Modules

Any module file you create should be saved into your local lmodfiles directory ($HOME/local/share/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

In the case of our Git example, you should create the directory $HOME/local/share/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/local/share/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using either module load git or module load git/2.9.0. 

If you install another version later on (lets say version 2.9.1) and want to created a module file for it, you need to make sure you call it 2.9.1.lua. When loading Git, lmod will automatically load the newer version. If you need to go back to an older version, you can do so by specifying the version you want: module load git/2.9.0.

To make sure you have the correct module file loaded, type which git which should give the output "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

To make sure the software was installed correctly and that the module is working, type git --version which shuold give the output "git version 2.9.0".

Further Reading

For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.

This "how to" will demonstrate how to lower ones' disk space usage.  The following procedures can be applied to all of OSC's file systems.

We recommend users regularly check their data usage and clean out old data that is no longer needed.

Users who need assistance lowering their data usage can contact OSC Help.

Preventing Excessive Data Usage Before It Starts

Users should ensure that their jobs are written in such a way that temporary data is not saved to permanent file systems, such as the project space file system or their home directory.  
 

If your job copies data from the scratch file system or its node's local disk ($TMPDIR) back to a permanent file system, such as the project space file system or a home directory, you should ensure you are only copying the files you will need later.

Identifying Old and Large Data

The following commands will help you identify old data using the find command.

Find all files in a directory that have not been accessed in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not accessed in the past 100 days.  

The last access time atime is updated when a file is opened by any operation, including grep , cat , head , sort , etc.

find ~ -atime +100 -exec ls -l {} \;

• To search a different directory replace ~ with the path you wish to search.  A period . can be used to search the current directory.
• To view files not accessed over a different time span, replace 100 with your desired number of days.
• To view the total size in bytess of all the files found by find , you can add  | awk '{s+=$5} END {print "Total SIZE (bytes): " s}' to the end of the command:

find ~ -atime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

Find all files in a directory that have not been modified in the past 100 days:

This command will recursively search the users home directory and give a detailed listing of all files not modified in the past 100 days.

The last modified time mtime is updated when a file's contents are updated or saved.  Viewing a file will not update the last modified time.

find ~ -mtime +100 -exec ls -l {} \; 

• To search a different directory replace  ~  with the path you wish to search.  A period  .  can be used to search the current directory.
• To view files not modified over a different time span, replace  100  with your desired number of days.
• To view the total size in bytes of all the files found by find , you can add  | awk '{s+=$5} END {print "Total SIZE (bytes): " s}'  to the end of the command:

find ~ -mtime +100 -exec ls -l {} \;| awk '{s+=$5} END {print "Total SIZE (bytes): " s}'

List files larger than a specified size:

Adding the -size <size>  option and argument to the find command allows you to only view files larger than a certain size.  This option and argument can be added to any other find command.

For example, to view all files in a users home directory that are larger than 1GB:

find ~ -size 1G -exec ls -l {} \;

Deleting Identified Data

If you no longer need the old data, you can delete it using the rm command.

If you need to delete a whole directory tree (a directory and all of its subcontents, including other directories), you can use the rm -R command.  

For example, the following command will delete the data directory in a users home directory:

rm -R ~/data

If you would like to be prompted for conformation before deleting every file, use the -i option.

rm -Ri ~/data 

Enter y or n when prompted.  Simply pressing the enter button will default to n .

Deleting files found by find

rm can be combined with any find command to delete the files found.  The syntax for doing so is:

find <location> <other find options> -exec rm -i {} \;

Where <other find options> can include one or more of the options  -atime <time> , -mtime <time> , and -size <size> .

The following command would find all files in the ~/data directory 1G or larger that have not been accessed in the past 100 days, and then prompt for confirmation to delete each file:

find ~/data -atime +100 -size 1G -exec rm -i {} \;

If you are absolutely sure the files identified by find are okay to delete you can remove the -i option to rm and you will not be prompted.  Extreme caution should be used when doing so!

Archiving Data

If you still need the data but do not plan on needing the data in the immediate future, contact OSC Help to discuss moving the data to a archive file system.  Requests for data to be moved to the archive file system should be larger than 1TB. 

Compressing

If you need the data but do not access the data frequently, you should compress the data using tar or gzip.

Moving Data to a Local File System

If you have the space available locally you can transfer your data there using sftp or Globus. 

Globus is recommended for large transfers.

The OnDemand File application should not be used for transfers larger than 1GB.

Often users want to submit a large amount of jobs all at once with each using different parameters for each job.  These parameters could be anything, including the path of a data file or different input values for a program.  This how-to will show you how you can do this using a simple python script, a CSV file, and a template script.  You will need to adapt this advice for your own situation.

Consider the following batch script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n week42_data8

# Copy input data to the nodes fast local disk 
cp ~/week42/data/source1/data8.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis data8.in data8.out

# Copy results to proper folder
cp  data8.out ~/week42/results

Lets say you need to submit 100 of these jobs on a weekly basis.  Each job uses a different data file as input.  You recieve data from two different sources, and thus your data is located within two different folders.  All of the jobs from one week need to store their results in a single weekly results folder.  The output file name is based upon the input file name.

Creating a Template Script

As you can see, this job follows a general template.  There are three main parameters that change in each job:

1. The week 
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to copy the results to the correct folder
2. The data source
   • Used to find the proper data file to copy to the nodes local disk
3. The data file's name
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to specify both the input and output file to the program full_analysis
   • Used to copy the results to the correct folder

If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n ${WEEK}_${DATA}

# Copy input data to the nodes fast local disk 
cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis ${DATA}.in ${DATA}.out

# Copy results to proper folder
cp  ${DATA}.out ~/${WEEK}/results

Automating Job Submission

We can now use qsub's -v option to pass parameters to our template script.  The format for passing parameters is:

qsub -v par_name=par_value[,par_name=par_value...] script.sh

Submitting a 100 jobs using qsub -v option manually does not make our task much easier than modifying and submitting each job one by one.  To complete our task we need to automate the submission of our jobs.  We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.

Here is the script that submits the jobs using the parameters:

#!/usr/bin/env python
import csv, subprocess

parameter_file_full_path = "/nfs/12/user0123/week42/job_params.csv"

with open(parameter_file_full_path, "rb") as csvfile:
    reader = csv.reader(csvfile)
    for job in reader:
        qsub_command = """qsub -v WEEK={0},SOURCE={1},DATA={2} template_1.pbs""".format(*job)

        #print qsub_command # Uncomment this line when testing to view the qsub command

        # Comment the following 3 lines when testing to prevent jobs from being submitted
        exit_status = subprocess.call(qsub_command, shell=True)
        if exit_status is 1:  # Check to make sure the job submitted
            print "Job {0} failed to submit".format(qsub_command)
print "Done submitting jobs!"

This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values.  If the qsub command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display.  The jobs will be submitted using the general format:

qsub -v WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.pbs

Where WEEK_VALUE ,SOURCE_VALUE, and DATA_VALUE are the first, second, and third comma separated values in the CSV file's current row, and template_1.pbs is the name of our template script.

Creating a CSV File

We now need to create a CSV file with parameters for each job.  This can be done with a regular text editor or using a spreadsheet editor such as excel.  By default you should use commas as your delimiter.  

Here is our CSV file with parameters:

week42,source1,data1
week42,source1,data2
week42,source1,data3
...
week42,source2,data98
week42,source2,data99
week42,source2,data100

The submit script would read in the first row of this CSV file and form and  submit the following qsub command:

qsub -v WEEK=week42,SOURCE=source1,DATA=data1 template_1.pbs

Submitting Jobs

Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.  

Globus Connect is a reliable, high-performance file transfer platform allowing users to transfer large amounts of data seamlessly between systems.  It aims to make transfers a "click-and-forget" process by setting up configuration details in the background and automating fault recovery.  

Globus can be used for both file transfers between OSC and:

• A computing institution with Globus installed (check with your site provider for availability) or
• A personal computer (known as a personal endpoint)

Users transferring between OSC and another computing institution with​ Globus installed do not need to install Globus Connect Personal, and can skip to Usage.

More on how Globus works can be found on the Globus "How It Works" page.

Install Globus Connect Personal

To use Globus to transfer from a personal computer, you will need to install the Globus Connect Personal client on your computer following the steps below. Those transferring between OSC and another computing institution can skip to Usage.

1. Sign up for a free Globus account
2. Download the Globus Connect Personal Client 
   • Click "Manage Endpoints" under the "Manage Transfers/Data" menu
   • Click "add Globus Connect" on the top-right of the page
   • Choose a unique name for your endpoint and generate the setup key
   • Download the Globus Connect client for your operating system
3. Install Globus Connect Personal Client
   • Windows
     1. Run the Installer
     2. Copy-Paste the setup key to complete the installation
   • Mac​​
     1. Mount your drives
     2. Copy the Globus Client to your application Folder
     3. Start The Globus Client, and enter the provided setup key
   • Linux
     1. Un-tar the .tgz file with the command tar -zxvf
     2. Run globusconnect , found within the unzipped directory
     3. Copy-Paste the setup key when prompted
4. (Optional) Changing directories accessible to Globus

By default Globus will only add certain default folders to the list of files and directories accessible by Globus. To change/add/remove files and directories from this list:

Windows

1. Start Globus Connect Personal
2. Go to "Options"

• Add directories/files using the  "+" button
• Remove directories/files using the "-" button
• Revert to the default accessible directories/files using the "Reset to Defaults" button
• Any changes you make are not made permanent until you press the "Save" button

​​Mac

1. Start Globus Connect Personal
2. Go to Preferences -> Access

• Add directories/files using the  "+" button
• Remove directories/files using the "-" button
• Revert to the default accessible directories/files using the "Reset to Defaults" button
• Any changes you make are not made permanent until you press the "Save" button

​Linux​

1. Configure the   ~/.globusonline/lta/config-paths  file. This file is a plain text file, with each line corresponding to the configuration of a particular directory path you wish to make accessible. Each line consists of 3 comma separated fields as below

   <path1>,<sharing flag>,<R/W flag>
   <path2>,<sharing flag>,<R/W flag>
   <path3>,<sharing flag>,<R/W flag>
   ...
   

   Path: an absolute directory/path to be permitted. A leading tilde "~" can be used to represent the home directory of the user that runs globusconnectpersonal.

   Sharing Flag: it controls sharing, with a value of "1" allowing sharing for the path and a value of "0" disallowing sharing for the path.

   R/W Flag: it determines whether the path will be accessible read-only or read/write, with a "1" permitting read/write access and a "0" marking the path as read-only. 

2. After making a change to the ~/.globusonline/lta/config-paths file, you must stop and restart Globus Connect Personal before the changes will take effect as below

$ ./globusconnectpersonal -stop
$ ./globusconnectpersonal -start &

Usage

1. Login to Globus and navigate to the "Transfer Files" under the "Manage Data" menu
2. Enter your end point in one of the boxes
   • If transferring to a computer with Globus Connect Personal installed, this will be the unique name chosen during installation. It is also listed under "Administered by Me"
3. Enter osc#gcs (OSC Globus Connect Server) as the other endpoint
   • Enter your OSC username and password for the authetication
4. You can now transfer files and directories both ways by selecting them and pressing the arrow indicating which way you'd like to transfer

Further Reading

• https://www.globus.org/
• www.youtube.com/GlobusOnline

It is now possible to run Docker and Singularity containers on the Owens cluster at OSC. Single-node jobs are currently supported; MPI jobs and GPU jobs are planned for the future.

From the Docker website:  "A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings."

This document will describe how to run Docker and Singularity containers on Owens. You can use containers from Docker Hub, Singularity Hub, or any other source. As examples we will use vsoch/singularity-hello-world and ubuntu from Docker Hub.

If you want to create your own container or modify a container, you will have to do it on a system where you have root access. You cannot build or update a container on Owens or any other OSC system.

Note:  This “how-to” is a work in progress!

Setting up your environment for Singularity usage

To load the Singularity environment on Owens, run the following commands:

module use /usr/local/share/lmodfiles/project/osc
module load singularity

Accessing a container

A Singularity container is a single file with a .img extension.

You can either download (“pull”) a container from a hub or run it directly from the hub. Or just copy the container from anywhere.

Examples of “pulling” a container:

Example:  Pull from the 7.2.0 branch of the gcc repository on Docker Hub. The 7.2.0 is called a tag.

singularity pull docker://gcc:7.2.0

Filename:  gcc-7.2.0.img

Example:  Pull the vsoch/singularity-hello-world container from the Singularity hub. Since no tag is specified it pulls from the master branch of the repository.

singularity pull shub://vsoch/hello-world

Filename:  vsoch-hello-world-master.img

Example:  Pull an Ubuntu container from Docker Hub.

singularity pull docker://ubuntu

Filename:  ubuntu.img

Running a container

There are four ways to run a container under Singularity.

You can do all this either in a batch job or on a login node. (Don’t run on a login node if the container will be performing heavy computation, of course.)

We note that the operating system on Owens is Red Hat:

[owens-login01]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"

In the examples below we will often check the operating system to show that we are really inside a container.

Run container like a native command

If you simply run the container image it will execute the container’s runscript.

Example:  Run an Ubuntu shell. (See also the “shell” sub-command below.)

[owens-login01]$ ./ubuntu.img
[owens-login01]$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
[owens-login01]$ exit
exit
[owens-login01]$

Example:  Run vsoch/hello-world

Note that this container returns you to your native OS after you run it.

[owens-login01]$ ./vsoch-hello-world-master.img
RaawwWWWWWRRRR!!
[owens-login01]$

Use the “run” sub-command

The Singularity “run” sub-command does the same thing as running a container directly as described above. That is, it executes the container’s runscript.

Example:  Run a container from a local file

[owens-login01]$ singularity run vsoch-hello-world-master.img
RaawwWWWWWRRRR!!
[owens-login01]$

Example:  Run a container from a hub without explicitly downloading it

[owens-login01]$ singularity run shub://vsoch/hello-world
Progress |===================================| 100.0%
RaawwWWWWWRRRR!!
[owens-login01]$

Use the “exec” sub-command

The Singularity “exec” sub-command lets you execute an arbitrary command within your container instead of just the runscript.

Example:  Find out what operating system the vsoch/hello-world container uses

[owens-login01]$ singularity exec vsoch-hello-world-master.img cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
[owens-login01]$ 

Use the “shell” sub-command

The Singularity “shell” sub-command invokes an interactive shell within a container.

Example:  Examine the contents of the vsoch/hello-world container

Note:  A container’s runscript is the file /singularity within the container.

[owens-login01]$ singularity shell vsoch-hello-world-master.img
Singularity: Invoking an interactive shell within container...
Singularity vsoch-hello-world-master.img:~/singularity> ls /
bin   dev  home  lib64       media  opt   rawr.sh  run   singularity  sys  users  var
boot  etc  lib   lost+found  mnt    proc  root     sbin  srv          tmp  usr
Singularity vsoch-hello-world-master.img:~/singularity> cat /singularity
#!/bin/sh
exec /bin/bash /rawr.sh
Singularity vsoch-hello-world-master.img:~/singularity> cat /rawr.sh
#!/bin/bash
echo "RaawwWWWWWRRRR!!"
Singularity vsoch-hello-world-master.img:~/singularity> exit
exit
[owens-login01]$

Example:  Run an Ubuntu shell. Note the “Singularity” prompt within the shell.

[owens-login01]$ singularity shell ubuntu.img
Singularity: Invoking an interactive shell within container...

Singularity ubuntu.img:~/singularity> cat /singularity
#!/bin/sh

exec /bin/bash "$@"Singularity ubuntu.img:~/singularity> exit
exit
[owens-login01]$

File system access

When you use a container you run within the container’s environment.  The only directories available to you by default from the host environment (Owens) are

• your home directory
• working directory (directory you were in when you ran the container)
• /tmp

If you want access to scratch or project directories you must bind them at the time you run the container. This option applies to the run, exec, and shell sub-commands. The format is -B or --bind followed by the path on the host, “:”, bind point in the container.

Example:  Bind my scratch and project directories

singularity shell -B /fs/scratch/username:/scratch -B /fs/project/MYPROJ:/project ubuntu.img

Inside the container your scratch and project directories will be available at /scratch and /project respectively.

Example:  Bind /fs, which includes all scratch and project directories.

singularity shell -B /fs ubuntu.img

This lets you keep your familiar paths. The “:” and bind point aren’t needed because the same name is being used for the container and the host.

References

• http://singularity.lbl.gov/user-guide
• https://hpc.nih.gov/apps/singularity.html
• https://www.docker.com/

This document shows you how to use the NFSv4 ACL permissions system. An ACL is a list of permissions associated with a file or directory. These permissions allow you to restrict access to a certian file or directory by user or group. NFSv4 ACLs provide more specific options than typical POSIX read/write/execute permissions used in most systems.

Understanding NFSv4 ACL

This is an example of an NFSv4 ACL

A::user@nfsdomain.org:rxtncy

A::alice@nfsdomain.org:rxtncy

A::alice@nfsdomain.org:rxtncy

A::alice@nfsdomain.org:rxtncy

The following sections will break down this example from left to right and provide more usage options

ACE Type

The 'A' in the example is known as the ace type. The 'A' denotes "Allow" meaning this ACL is allowing the user or group to perform actions requiring permissions. Anything that is not explicitly allowed is denied by default.

ACE Flags

The above example could have a distinction known as a flag shown below

A:d:user@osc.edu:rxtncy

The 'd' used above is called an inheritence flag. This makes it so the ACL set on this directory will be automatically established on any new subdirectories. Inheritence flags only work on directories and not files. Multiple inheritence flags can be used in combonation or omitted entirely. Examples of inheritence flags are listed below:

ACE Principal

The 'user@nfsdomain.org' is a principal. The principle denotes the people the ACL is allowing access to. Principals can be the following:

• A named user
  • Example: user@nfsdomain.org
• Special principals
  • OWNER@
  • GROUP@
  • EVERYONE@
• A group
  • Note: When the principal is a group, you need to add a group flag, 'g', as shown in the below example

  • A:g:group@osc.edu:rxtncy
    

ACE Permissions

The 'rxtncy' are the permissions the ACE is allowing. Permissions can be used in combonation with each other. A list of permissions and what they do can be found below:

Note: Aliases such as 'R', 'W', and 'X' can be used as permissions. These work simlarly to POSIX Read/Write/Execute. More detail can be found below.

Using NFSv4 ACL

This section will show you how to set, modify, and view ACLs

Set and Modify ACLs

To set an ACE use this command:

nfs4_setfacl [OPTIONS] COMMAND file

To modify an ACE, use this command:

nfs4_editfacl [OPTIONS] file

Where file is the name of your file or directory. More information on Options and Commands can be found below.

Options

Options can be used in combination or ommitted entirely. A list of options is shown below:

Commands

Commands are only used when first setting an ACE. Commands and their uses are listed below.

View ACLs

To view ACLs, use the following command:

nfs4_getfacl file

Where file is your file or directory

SSHing directly to a compute node at OSC - even if that node has been assigned to you in a current batch job - and starting VNC is an "unsafe" thing to do. When your batch job ends (and the node is assigned to other users), stray processes will be left behind and negatively impact other users. However, it is possible to use VNC on compute nodes safely.

The examples below are for Oakley. If you use other systems, please see this page for supported versions of TurboVNC on our systems.

Starting your VNC server

Step one is to create your VNC server inside a batch job.

Option 1: Interactive

The preferred method is to start an interactive job, requesting an entire node, and then once your job starts, you can start the VNC server.

qsub -I -l nodes=1:ppn=12:gpus=2:vis

This command requests an entire GPU node, and tells the batch system you wish to use the GPUs for visualization. This will ensure that the X11 server can access the GPU for acceleration. In this example, I have not specified a duration, which will then default to 1 hour.

module load virtualgl
module load turbovnc/2.0

Then start your VNC server. (The first time you run this command, it may ask you for a password - this is to secure your VNC session from unauthorized connections. Set it to whatever password you desire. We recommend a strong password.)

vncserver

The output of this command is important: it tells you where to point your client to access your desktop. Specifically, we need both the host name (before the :), and the screen (after the :).

New 'X' desktop is n0302.ten.osc.edu:1

Option 2: Batch

This option is less optimal, because it is slightly more difficult to get the hostname and screen. However, by submitting a non-interactive batch job, you can go away and have the system email you when your desktop is ready to be connected to, and more importantly if your SSH connection to OSC is somewhat unstable and intermittent, you do not run the risk of being disconnected during your interactive session and having your VNC server terminated. In general, it is recommended you only use this option if running via an interactive session is not feasible.

In order to start an VNC session non-interactively, you can submit the following script to the scheduler using qsub (adjusting your walltime to what you need):

#PBS -l nodes=1:ppn=12:gpus=2:vis
#PBS -l walltime=00:15:00
#PBS -m b
#PBS -N VNCjob
#PBS -j oe
module load virtualgl
module load turbovnc/2.0
vncserver
sleep 100
vncpid=`pgrep -s 0 Xvnc`
while [ -e /proc/$vncpid ]; do sleep 0.1; done

This script will send you an email when your job has started, which includes the hostname.

PBS Job Id: 9200155.oak-batch.osc.edu
Job Name:   VNCjob
Exec host:  n0311/0-11
Begun execution

The screen is virtually always "1", unless someone else started a VNC server on that node outside of the batch system. You can verify the output of the vncserver command by using qpeek on a login node:

qpeek jobid

Where "jobid" is the batch system job number, for example, "9200155".

Connecting to your VNC server

Because the compute nodes of our clusters are not directly accessible, you must log in to one of the login nodes and allow your VNC client to "tunnel" through SSH to the compute node. The specific method of doing so may vary depending on your client software.

Linux/MacOS

Option 1: Manually create an SSH tunnel 

I will be providing the basic command line syntax, which works on Linux and MacOS. You would issue this in a new terminal window on your local machine, creating a new connection to Oakley.

ssh -L 5901:n0302.ten.osc.edu:5901 guilfoos@oakley.osc.edu

Open your VNC client, and connect to "localhost:1" - this will tunnel to the correct node on Oakley.

Option 2: Use your VNC software to tunnel 

This example uses Chicken of the VNC, a MacOS VNC client.

The default window that comes up for Chicken requires the host to connect to, the screen (or port) number, and optionally allows you to specify a host to tunnel through via SSH. This screenshot shows a proper configuration for the output of vncserver shown above. Substitute your host, screen, and username as appropriate.

When you click [Connect], you will be prompted for your HPC password (to establish the tunnel, provided you did not input it into the "password" box on this dialog), and then (if you set one), for your VNC password. If your passwords are correct, the desktop will display in your client.

Windows

This example shows how to create a SSH tunnel through your ssh client.  We will be using Putty in this example, but these steps are applicable to most SSH clients.

First, make sure you have x11 forwarding enabled in your SSH client.

Next, open up the port forwarding/tunnels settings and enter the hostname and port you got earlier in the destination field.  You will need to add 5900 to the port number when specifiying it here.  Some clients may have separate boxes for the desination hostname and port.  

For source port, pick a number between 11-99 and add 5900 to it.  This number between 11-99 will be the port you connect to in your VNC client.

Make sure to add the forwaded port, and save the changes you've made before exiting the configutations window.

Now start a SSH session to the respective cluster your vncserver is running on.  The port forwarding will automatically happen in the background.  Closing this SSH session will close the forwarded port; leave the session open as long as you want to use VNC.

Now start a VNC client.  TurboVNC has been tested with our systems and is recommended.  Enter localhost:[port], replacing [port] with the port between 11-99 you chose earlier.

If you've set up a VNC password you will be prompted for it now.  A desktop display should pop up now if everything is configured correctly.

How to Kill a VNC session?

Occasionally you may make a mistake and start a VNC server on a login node or somewhere else you did not want to.  In this case it is important to know how to properly kill your VNC server so no processes are left behind.

The command syntax to kill a VNC session is:

vncserver -kill :[screen]

In the example above, screen would be 1.

You need to make sure you are on the same node you spawned the VNC server on when running this command.

Many software packages require a license.  These licenses are usually made available via a license server, which allows software to check out necessary licenses.  In this document external refers to a license server that is not hosted inside OSC.

If you have such a software license server set up using a license manager, such as FlexNet, this guide will instruct you on the necessary steps to connect to and use the licenses at OSC.

Users who wish to host their software licenses inside OSC should consult OSC Help.

Introduction

Broadly speaking, there are two different ways in which the external license server's network may be configured.  These differ by whether the license server is directly externally reachable or if it sits behind a private internal network with a port forwarding firewall.  

If your license server sits behind a private internal network with a port forwarding firewall you will need to take additional steps to allow the connection from our systems to the license server to be properly routed. 

License Server is Directly Externally Reachable

License Server is Behind Port Forwarding Firewall

Unsure?

If you are unsure about which category your situation falls under contact your local IT administrator.

Configure Remote Firewall

In order for connections from OSC to reach the license server, the license server's firewall will need to be configured.  All outbound network traffic from all of OSC's compute nodes are routed through a network address translation host (NAT), or two backup servers.

The license server should be configured to allow connections from the following IP addresses to the SERVER:PORT where the license server is running:

• nat.osc.edu (192.157.5.13)
• 192.148.248.35
• 192.148.248.186

A typical FlexNet-based license server uses two ports: one is server port and the other is demon port, and the firewall should be configured for the both ports. A typical license file looks, for example,

SERVER licXXX.osc.edu 0050XXXXX5C 28000
VENDOR {license name} port=28001

In this example, "28000" is the server port, and "28001" is the demon port. The demon port is not mandatory if you use it on a local network, however it becomes necessary if you want to use it outside of your local network. So, please make sure you declared the demon port in the license file and configured the firewall for the port.

Confirm Configuration

The firewall settings should be verified by attempting to connect to the license server from the compute environment using telenet.

Get on to a compute node by requesting a short, small, interactive job and test the connection using telenet:

telnet <License Server IP Address> <Port#>

(Recommended) Restrict Access to IPs/Usernames

It is also recommended to restrict accessibility using the remote license server's access control mechanisms, such as limiting access to particular usernames in the options.dat file used with FlexNet-based license servers.

For FlexNet tools, you can add the following line to your options.dat file, one for each user.

INCLUDEALL USER <OSC username>

If you have a large number of users to give access to you may want to define a group using GROUP within the options.dat file and give access to that whole group using INCLUDEALL GROUP <group name> .

Users who use other license managers should consult the license manager's documentation.

Modify Job Environment to Point at License Server

The software must now be told to contact the license server for it's licenses.  The exact method of doing so can vary between each software package, but most use an environment variable that specifies the license server IP address and port number to use.

For example LS DYNA uses the environment variable LSTC_LICENSE and LSTC_LICENSE_SERVER to know where to look for the license.  The following lines would be added to a job script to tell LS-DYNA to use licenses from port 2345 on server 1.2.3.4:

setenv LSTC_LICENSE network
setenv LSTC_LICENSE_SERVER 2345@1.2.3.4

License Server is Behind Port Forwarding Firewall

If the license server is behind a port forwarding firewall, and has a different IP address from the IP address of the firewall, additional steps must be taken to allow connections to be properly routed within the license server's internal network.  

1. Use the license server's fully qualified domain name in the environment variables instead of the IP address.
2. Contact OSC Help to have the firewall IP address mapped to the fully qualified domain name.

Software Specific Details

The following outlines details particular to a specific software package.  

ANSYSLI_SERVERS=<port>@<IP>
ANSYSLMD_LICENSE=<port>@<IP>

This document shows you how to set soft limits using ulimit command.

The ulimit command sets or reports user process resource limits. The default limits are defined and applied when a new user is added to the system. Limits are categorized as either soft or hard. With the ulimit command, you can change your soft limits for the current shell environment, up to the maximum set by the hard limits. You must have root user authority to change resource hard limits.

Syntax

ulimit [-H] [-S] [-a] [-c] [-d] [-f] [-m] [-n] [-s] [-t] [Limit]

The limit for a specified resource is set when the Limit parameter is specified. The value of the Limit parameter can be a number in the unit specified with each resource, or the value "unlimited". For example, to set the file size limit to 51,200 bytes, use:

ulimit -f 100

To set the size of core dumps to unlimited, use:

ulimit –c unlimited

How to change ulimit for a MPI program

ulimit command affects the current shell environment. When a MPI program is started, it does not spawn in the current shell. You have to use mpiexec to start a wrapper script that sets the limit if you want to set the limit for each process. Below is how you set the limit for each shell (We use  ulimit –c unlimited to allow unlimited core dumps, as an example): 

1. Prepare your batch job script named "myjob" as below (Here, we request 5-hour 2-node on Oakley cluster):

#PBS -l nodes=2:ppn=12
#PBS -l walltime=5:00:00
#PBS ...

#!/bin/bash

cd $PBS_O_WORKDIR
...
mpiexec ./test1
...

2. Prepare the wrapper script named "test1" as below:

#!/bin/bash
ulimit –c unlimited
.....(your own program)

3. qsub myjob

This page outlines a way a professor can set up a file submission system at OSC for his/her classroom project.

Usage for Professor

After connect to OSC system, professor runs submit_prepare as

$ /fs/scratch/xwang/bin/submit_prepare

Follow the instructions. It will create designated directory where students submit their assignments, as well as generate submit for students used to submit homework to OSC, both of which are located in the directory specified by the professor.

Note:

The PI can also enforce deadline by simply changing the permission of the submission directory or renaming the submission directory at the deadline.

(Only works on Owens): One way is to use at command following the steps below:

• Use at command to specify the deadline:

at [TIME]

where TIME is formatted HH:MM AM/PM MM/DD/YY. For example:

at 2:30 PM 08/21/2017

• After running this command, run:

$ chmod 700 [DIRECTORY]

where DIRECTORY is the assignment folder to be closed off.

• Enter [ctrl+D] to submit this command.

The permission of DIRECTORY will be changed to 700 at 2:30PM, August 21, 2018. After that, student will get an error message when he/she tries to submit assignment to this directory.

Usage for Students

Student should create one directory which includes all the files he/she wants to submit before running this script to sumit his/her assignment. Also, the previous submission of the same assignment from the student will be replaced by the new submission. 

To submit the assignemt, student runs submit after connecting to OSC system as

$ /path/to/directory/from/professor/submit

Follow the instructions. It will allow students to submit assignment to designated directory speficifed by professor and send a confirmation email, or return error message.