When you connect to one of the HPC systems at OSC, you are working on a login node. This document describes the computing environment on the login nodes.
Summary of OSC-specific information
If you are familiar with Linux, please at least read this section.
- Login nodes have a shell limit of 20 minutes CPU time. Use the batch system for serious computing. For large file transfers use sftp.osc.edu instead of a login node.
- Each HPC system at OSC has multiple login nodes. Home directories are visible everywhere.
- OSC projects correspond to Linux groups.
- OSC uses a module system for setting up the software environment. Oakley and Glenn use different module systems with slightly different commands.
When you first log in, you’ll see two important pieces of information. The first is the message of the day. The message of the day includes important system notices such as scheduled downtime. These notices are also posted on https://www.osc.edu/. You can display the motd again with the command
The second piece of information displayed is information about your disk usage and quota. You have at least one and possibly more allocations of disk space associated with your user name. Each allocation has limits on the amount of space and the number of files you can use. This information is collected nightly and displayed when you log in. If you exceed your quota, further commands and jobs will fail, so keep an eye on it. You can display your current usage with the
The login nodes, like the compute nodes, run the Linux operating system. You will be greeted with a prompt, for example:
This prompt comes from the shell, which is the command-line interpreter for Linux.
exit command terminates your login session.
Sample login session where the user does nothing except
Last login: Mon Jun 25 16:02:13 2012 from 18.104.22.168 ****************************************************************************** Welcome to the Ohio Supercomputer Center! PLEASE SEE THE OSC TECHNICAL INFORMATION SERVER (http://www.osc.edu/hpc) FOR ANNOUNCEMENTS/NOTICES AND USER INFORMATION. For questions or assistance, contact firstname.lastname@example.org. ***************************************************************************** 2012/05/01 All OSC systems are operating normally. ***************************************************************************** As of 2012 Jun 25 04:02 userid usr1234 on /nfs/14 used 250GB of quota 500GB and 560634 files of quota 1000000 files As of 2012 Jun 25 04:02 project/group G-2959 on /nfs/proj09 used 246GB of quota 2861GB and 1087715 files of quota 2000000 files -bash-4.1$ exit logout
Purpose of login nodes
The login nodes are the gateway to the HPC systems. They should never be used for serious computing. The login nodes are intended for tasks such as text editing, managing your files, submitting jobs to the batch system, and monitoring your jobs. They are also suitable for compiling and linking your code, unless the code is huge. Small-scale interactive work may also be acceptable if your resource requirements are minimal.
The login shell limits your computations to 20 minutes of CPU time. When you reach that limit your process will be killed with, unfortunately, no explanation. Remember: Computation should be done through the batch system, not on the login node.
Large file transfers sometimes hit the 20-minute limit because of the encryption overhead. See below for instructions on transferring large files.
Login node structure
Each HPC system at OSC has its own set of login nodes. Oakley has two login nodes; Glenn has four. Connections are done in a round-robin fashion. When you ssh to oakley.osc.edu you will actually be connected to either oakley01 or oakley02. For glenn.osc.edu the login nodes are opt-login01 through opt-login04.
If you are just working with your files and directories it doesn’t matter which system you log into. All the login nodes on both systems have all the same file systems mounted. If you are submitting or monitoring batch jobs or building code, it does matter which system you are logged into, but not which login node on that system. You can find out which node you are on with the
hostname command, although you rarely need to know.
Linux is a “Unix-like” operating system that is widely used in HPC. Your interaction with Linux is through a command-line shell, which displays a prompt and interprets the commands you type in.
You have a choice of several shells: bash, csh, tcsh, or ksh. The default shell on OSC systems is bash, but you can change it through the ARMSTRONG portal under the HPC Account tab. The different shells are similar in concept but differ in some of their commands and syntax. If you don’t have a strong preference, we suggest you stay with bash.
This document is not intended as a Linux/Unix tutorial; there are plenty of those available. We describe just a few commands that you can’t live without.
Throughout this documentation, we will use the dollar sign ($) as a generic prompt in our examples. User input is shown in bold; output is shown in normal type.
One of the most important commands is
man, which displays online manual pages. Most commands have a man page. For example, to see the man page for the
$ man mkdir
MKDIR(1) User Commands MKDIR(1) NAME mkdir - make directories SYNOPSIS mkdir [OPTION]... DIRECTORY... DESCRIPTION Create the DIRECTORY(ies), if they do not already exist. [ remainder of output omitted ] $
When you log in to any of the login nodes you’ll be working in your home directory. Each user has a home directory that is shared across all OSC systems. Whatever system you connect to, you’ll have the same home directory. You can create files and subdirectories in your home directory.
File permissions in Unix/Linux are based on the concepts of “user”, “group”, and “other”. “User” is just you. “Other”, or “world”, in our case means everyone with an OSC user ID. Your primary “group” at OSC corresponds to the project your user ID is associated with. Groups are also used to control access to licensed software. You can set different permissions on your files and directories for each of these categories. The permissions you can set are read, write, and execute. The default permissions on your home directory allow anyone with an OSC HPC account to read your files, although only you have write permission. You can change the permissions with the
chmod command if you want to restrict access.
You can transfer files between an outside computer and your home directory using any of these protocols: sftp, scp, rsync. These are all secure transfers, meaning that they encrypt your data. If you have large files, you may hit the 20-minute limit on CPU time on the login node. If this happens you may use sftp.osc.edu for your transfers instead of the login node. This server supports the same protocols as the login nodes, plus gridftp, but it doesn’t have the same shell limits. It is to be used only for file transfers.
Here are a few commands that you’re likely to use for managing your files. Look up the man pages for more information. Many other options are available. If this is new to you, be sure to read a Linux or Unix tutorial.
||List the files in the current working directory or specified directory. With the
||Change directory. If no directory is specified, change to the home directory. If a directory is specified, it can be either relative to the current directory or absolute. The “..” means the parent directory, up one level.|
||Print the full name of the current working directory.|
||Make a new directory in the current working directory.|
||Copy a file to a new location or a new name in the current directory.|
||Move a file to a new location or rename it in the current directory.|
||Display a file.|
||Display a file with paging.|
||Remove a file.|
||Make a file world readable (add “read” permission for “all”)
Make a file executable by me and my group (add “execute” permission for “user” and “group”)
Remove read and execute permissions for “other” from this directory and all files and directories below it, recursively.
||Save many files together into a single archive file; restore files from an archive. Compression options are available. Example is for archiving a directory.|
||Compress or uncompress a file.|
See the storage environment documentation for more information.
Graphics and GUIs
While Linux is primarily a command-line environment, many applications have graphical user interfaces (GUIs) based on a standard called X11. See the documentation for connecting to the OSC systems for instructions on getting graphics to work. A useful command for testing your X11 connection is
xclock, which displays a clock (with hands that move) if everything is working properly. Two things to check if it doesn’t work: 1) make sure you have an X server running on your local machine; and 2) make sure your ssh connection is set up to forward or tunnel X11. These steps are described in the documentation described above.
You can create and edit files using any of a number of text editors available in Linux. The traditional Unix editors vi and emacs are available; there are many books and web pages about vi and emacs if you’re interested.
A simpler editor to get you started is nano, a non-GUI editor that is fairly self-explanatory. Just type
nano at the prompt.
If you prefer a GUI-based editor, you can use gedit. You must have X11 forwarding enabled and have an X11 server running on your system. (See other documentation.) Then type
gedit at the prompt.
Yet another option is to edit your files on your own system and transfer them back and forth to the HPC system. If you do this you should be aware that text files created on Windows, Mac, and Linux all have different line endings, sometimes leading to difficult-to-diagnose problems.
Note to Windows users
The following command will convert your Windows text file to Unix format:
For conversion in the other direction:
On your Windows system, Wordpad will correctly display Unix files while Notepad will not.
Note to Mac users
The following command will convert your Mac text file to Unix format:
Your environment in Linux is controlled by a set of environment variables. Some of these variables are automatically set for you; others you control either directly or by loading environment modules. Environment variables are all upper case. To use their values you precede the name with a $.
A few of the environment variables you might need to know about are these:
||The absolute path of your home directory|
||List of directories where the system looks for programs to run|
||List of directories where the system looks for shared libraries to load|
To display the value of an environment variable, use the
echo command, for example:
$ echo $HOME /nfs/14/usr1234
(Recall that the $ at the beginning of the line in our examples represents the prompt and is not something that you type in.)
To go to your home directory:
You can set the value of an environment variable. The following examples set the value of
OMP_NUM_THREADS (an OpenMP parameter) to 12.
To set the value of an environment variable in bash:
To set the value of an environment variable in csh:
setenv OMP_NUM_THREADS 12
To run a program under Linux, type the program name followed by any arguments. If the program executable is not in your path, you must include the path with the program name.
Example of running a program that is in your path:
Example of running a program that is not in your path:
If you want to run a program located in your current directory, you should precede its name with “./”. The dot (“.”) represents your current directory. For security reasons, it is generally not part of your path. Example:
You can add a directory to your path, with the syntax depending on which shell you use.
Example in bash:
Example in csh:
setenv PATH $HOME/bin:$PATH
OSC has dozens of software packages installed. Rather than clutter up your environment with all of them, and probably running into conflicts as a result, we provide a system of modules. When you load a module, it modifies or sets the environment variables necessary for running the corresponding software. The module system is available in both the login environment and the batch environment. It is documented in “Batch Processing at OSC”.
Initializing your environment
When a shell is started, it reads commands from a set of startup files to initialize itself. OSC-provided files are executed first to set up the basic OSC environment, including the module system. If you have startup files in your home directory, they will be executed next. You can use this mechanism to set paths or load modules that you always use, for example.
The shell startup files are often called “dot files” because their names start with a period or “dot”. They don’t show up when you list a directory using the
ls command unless you add the option “
A bash login shell reads commands from
~/.bash_profile. A bash non-login interactive shell reads commands from
~/.bashrc. Many people have their
.bash_profile execute their
.bashrc to keep their login and non-login environments the same. The csh or tcsh shell reads
~/.tcshrc. (The symbol ~ refers to your home directory.)