GPU Computing

OSC offers GPU computing on all its systems.  While GPUs can provide a significant boost in performance for some applications the computing model is very different from the CPU.  This page will discuss some of the ways you can use GPU computing at OSC.

Accessing GPU Resources

To request nodes with a GPU add the gpus=# attribute to the PBS nodes directive in your batch script, for example, on Owens,

#PBS -l nodes=2:ppn=28:gpus=1

In most cases you'll need to load the cuda module (module load cuda) to make the necessary Nvidia libraries available. 

As of October 2019, there is no additional RU charge for GPUs for Ohio Academic clients.  There are charges for other clients such as commercial or non-profit.

Setting the GPU compute mode (optional)

The GPUs on Owens and Pitzer can be set to different compute modes as listed here.   They can be set by adding the following to the GPU specification: 

-l nodes=1:ppn=28:gpus=1:default
-l nodes=1:ppn=28:gpus=1:exclusive_process

The compute mode exclusive_process is the default on GPU nodes if a compute mode is not specified. With this mode,  mulitple CUDA processes across GPU nodes are not allowed, e.g CUDA processes via MPI.  If you need to run a MPI-CUDA job, please set the compute mode to  default

Using GPU-enabled Applications

We have several supported applications that can use GPUs.  This includes

Please see the software pages for each application.  They have different levels of support for multi-node jobs, cpu/gpu work sharing, and environment set-up.

Libraries with GPU Support

There are a few libraries that provide GPU implementations of commonly used routines. While they mostly hide the details of using a GPU there are still some GPU specifics you'll need to be aware of, e.g. device initialization, threading, and memory allocation.

MAGMA

MAGMA is an implementation of BLAS and LAPACK with multi-core (SMP) and GPU support. There are some differences in the API of standard BLAS and LAPACK.

cuBLAS and cuSPARSE

cuBLAS is a highly optimized BLAS from NVIDIA. There are a few versions of this library, from very GPU-specific to nearly transparent. cuSPARSE is a BLAS-like library for sparse matrices.

The MAGMA library is built on cuBLAS.

cuFFT

cuFFT is NVIDIA's Fourier transform library with an API similar to FFTW.

cuDNN

cuDNN is NVIDIA's Deep Neural Network machine learning library. Many ML applications are built on cuDNN.

Direct GPU Programming

GPUs present a different programming model from CPUs so there is a significant time investment in going this route.

OpenACC

OpenACC is a directives-based model similar to OpenMP. Currently this is only supported by the Portland Group C/C++ and Fortran compilers.

OpenCL

OpenCL is a set of libraries and C/C++ compiler extensions supporting GPUs (NVIDIA and AMD) and other hardware accelerators. The CUDA module provides an OpenCL library.

CUDA

CUDA is the standard NVIDIA development environment. In this model explicit GPU code is written in the CUDA C/C++ dialect, compiled with the CUDA compiler NVCC, and linked with a native driver program.

About OSC GPU Hardware

Our GPUs span several generations with different capabilites and ease-of-use. Many of the differences won't be visible when using applications or libraries, but some features and applications may not be supported on the older models.

 

Ruby K40

The K40 "Tesla" has a compute capability of 3.5, which is supported by most applications.

Each K40 has 12GB of memory and there is one GPU per GPU node.

Owens P100

The P100 "Pascal" is a NVIDIA GPU with a compute capability of 6.0. The 6.0 capability includes unified shared CPU/GPU memory -- the GPU now has its own virtual memory capability and can map CPU memory into its address space.

Each P100 has 16GB of on-board memory and there is one GPU per GPU node.

Pitzer V100

The V100 "Volta" is NVIDIA's flagship GPU with a compute capability of 7.0.

Each V100 has 16GB of memory and there are two GPUs per GPU node.

Examples

There are example jobs and code at GitHub