Migrating jobs from Glenn to Oakley or Ruby

This page includes a summary of differences to keep in mind when migrating jobs from Glenn to one of our other clusters.

Hardware

Most Oakley nodes have 12 cores and 48GB memory. There are eight large-memory nodes with 12 cores and 192GB memory, and one huge-memory node with 32 cores and 1TB of memory. Most Ruby nodes have 20 cores and 64GB of memory. There is one huge-memory node with 32 cores and 1TB of memory. By contrast, most Glenn nodes have 8 cores and 24GB memory, with eight nodes having 16 cores and 64GB memory.

Module System

Oakley and Ruby use a different module system than Glenn. It looks very similar, but it enforces module dependencies, and thus may prevent certain module combinations from being loaded that were permitted on Glenn. For example, only one compiler may be loaded at a time.

module avail will only show modules compatible with your currently loaded modules, but not all installed modules on the system. To see all modules on the cluster, use the command module spider. Both module avail and module spider can take a partial module name as a search parameter, such as module spider dyna.

Version numbers are indicated with a slash “/” rather than a dash “-” and need not be specified if you want the default version.

Compilers

Like Glenn, Oakley and Ruby support three compilers: Intel, PGI, and gnu. Unlike Glenn, Oakley and Ruby only let you have one compiler module loaded at any one time. The default is Intel. To switch to a different compiler, use module swap intel gnu or module swap intel pgi.

Important note: The gnu compilers are part of the Linux distribution, so they’re always available. It’s important to use the gnu module, however, to link with the correct libraries for MVAPICH, MKL, etc.

MPI

MPI-2 is available on Oakley and Ruby through the MVAPICH2 modules. The MVAPICH2 libraries are linked differently than on Glenn, requiring you to have the correct compiler and MVAPICH2 modules loaded at execution time as well as at compile time. (This doesn’t apply if you’re using a software package that was installed by OSC.)

Software you build and/or install

If your software uses any libraries installed by OSC, including MVAPICH, you will have to rebuild it. If you link to certain libraries, including MVAPICH, MKL, and others, you must have the same compiler module loaded at run time that you do at build time. Please refer to the compilation guide in our Knowledge Base for guidance on optimizing your compilations for our hardware.

OSC installed software

Most of the software installed on Glenn is also installed on Oakley or Ruby, although old versions may no longer be available. We recommend migrating to a newer version of the application if at all possible. Please review the software documentation to see what versions are available, and examine sample batch scripts.

Accounting

All OSC clusters currently use the same core-hour to RU conversion factor. Oakley will charge you for the number of cores proportional to the amount of memory your job requests, while Ruby only accepts full-node jobs. Please review the system documentation for each cluster.

“all” replaced by “pdsh”

The “all” command is not available on Oakley or Ruby; “pdsh” is available on all clusters.

pdsh –j jobid command

pdsh –g feature command

pdsh –w nodelist command

Service: