HOWTO: Locally Installing Software

Now that you have your directory structure created, you can install your software. For demonstration purposes, we will install a local copy of Git.

First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the wget tool to download it on the HPC. We'll download this into $HOME/osc_apps/git/2.9.0:

    mkdir -p $HOME/osc_apps/git/2.9.0
    cd $HOME/osc_apps/git/2.9.0
    wget https://github.com/git/git/archive/v2.9.0.tar.gz

Now extract the tar file:

    tar zxf v2.9.0.tar.gz

Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to install into $HOME/osc_apps/"software_name"/"version". Replace "software_name" with the software's name and "version" with the version you are installing, as demonstrated below. In this case, we'll use the configure tool's --prefix option to specify the install location.

You'll also want to specify a few variables to help make your application more compatible with our systems. For git, we recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add CC=icc CFLAGS=-static-intel to the end of your invocation of configure. If your application does not use configure, you can generally still set these variables somewhere in its Makefile or build script.

Then, we can build Git using the following commands:

    cd git-2.9.0
    make configure
    ./configure --prefix=$HOME/local/apps/git/2.9.0 CC=icc CFLAGS=-static-intel
    make && make install

Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.

Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths.

Automatically create a module

We can use the mkmod script to create a simple Lua module for the Git installation:

module load mkmod/2.2
export MODULEFILES_DIR=$HOME/osc_apps/lmodfiles
create_module.sh git 2.9.0 $HOME/osc_apps/git/2.9.0

It will create the module $HOME/osc_apps/lmodfiles/git/2.9.0.lua*. Please note that by default our mkmod script only creates module files that define some basic environment variables PATH, LD_LIBRARY_PATH, MANPATH, and GIT_HOME. These default variables may not cover all paths desired. We can overwrite these defaults in this way:

module load mkmod/2.2
export MODULEFILES_DIR=$HOME/osc_apps/lmodfiles
TOPDIR_LDPATH_LIST="lib:lib64" \
TOPDIR_PATH_LIST="bin:exe" \
create_module.sh git 2.9.0 $HOME/osc_apps/git/2.9.0

This adds $GIT_HOME/bin, $GIT_HOME/exe to PATH and $GIT_HOME/lib , $GIT_HOME/lib64 to LD_LIBRARY_PATH.

We can also add other variables by using ENV1, ENV2, and more. For example, suppose we want to change the default editor to vim for Git:

module load mkmod/2.2
export MODULEFILES_DIR=$HOME/osc_apps/lmodfiles
ENV1="GIT_EDITOR=vim" \
create_module.sh git 2.9.0 $HOME/osc_apps/git/2.9.0

* If you are using a different file structure for your local installs, you can override the environment variable MODULEFILES_DIR with the path to where you would like to store module files.

Manually create a module 

We will be using the filename 2.9.0.lua ("version".lua). A simple Lua module for our Git installation would be:

-- Local Variables
local name = "git"
local version = "2.9.0"

-- Locate Home Directory
local homedir = os.getenv("HOME")
local root = pathJoin(homedir, "local/apps", name, version)

-- Set Basic Paths
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("LD_LIBRARY_PATH", root .. "/lib")
prepend_path("LIBRARY_PATH", root .. "/lib")
prepend_path("INCLUDE", root .. "/include")
prepend_path("CPATH", root .. "/include")
prepend_path("PKG_CONFIG_PATH", root .. "/lib/pkgconfig")
prepend_path("MANPATH", root .. "/share/man")

This should be placed in $HOME/osc_apps/lmodfiles/git.

NOTE: For future module files, copy our sample modulefile from ~support/doc/modules/sample_module.lua. This module file follows the recommended design patterns laid out above and includes samples of many common module operations

Initializing Modules

Any module file you create should be saved into your local lmodfiles directory ($HOME/osc_apps/lmodfiles). To prepare for future software installations, create a subdirectory within lmodfiles named after your software and add one module file to that directory for each version of the software installed.

In the case of our Git example, you should create the directory $HOME/osc_apps/lmodfiles/git and create a module file within that directory named 2.9.0.lua.

To make this module usable, you need to tell lmod where to look for it. You can do this by issuing the command module use $HOME/osc_apps/lmodfiles in our example. You can see this change by performing module avail. This will allow you to load your software using  module load git/2.9.0. 

If you install another version later on (let's say version 2.9.1) and want to create a module file for it, you need to make sure you call it 2.9.1.lua.

To make sure you have the correct module file loaded, type which git which should emit "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

To make sure the software was installed correctly and that the module is working, type git --version which should emit "git version 2.9.0".

Simplified versions of the scripts used to manage the central OSC software installations are provided at ~support/share/install-script. The idea is that you provide the minimal commands needed to obtain, compile, and install the software (usually some variation on wget, tar, ./configure, make, and make install) in a script, which then sources an OSC-maintained template that provides all of the "boilerplate" commands to create and manage a directory structure similar to that outlined in the Getting Started section above. This is prefferred to manual installation since the installation and module file creation can easily be repeated and modified for new versions of software. You can copy an example install script from ~support/share/install-script/install-osc_sample.sh and follow the notes in that script, as well as in ~support/share/install-script/README.md, to modify it to install software of your choosing. For demonstration purposes we will be installing a local copy of git.

Creating the install script

For the git installation, begin by creating the directory osc_apps/git/install-script and copying the install script to it.

mkdir -p $HOME/osc_apps/git/install-script
cp ~support/share/install-script/install-osc_sample.sh $HOME/osc_apps/git/2.9.0/install-osc.sh

Next, open the copied install script with your text editor of choice. Following the steps in the manual installation section and the comments in install-osc_sample.sh, we can fill out the install script with our commands to download and build git. As a reminder, here are what the manual steps look like:

# (Obtain source step) Download source code into the download directory
cd $HOME/osc_apps/git/2.9.0
wget https://github.com/git/git/archive/v2.9.0.tar.gz

# (Setup step) Extract the tar file
tar zxf v2.9.0.tar.gz

# (Configure step) Make and run the configure file in the build directory
# Note: the script uses build directory BUILD instead of git-2.9.0
cd git-2.9.0
make configure
./configure --prefix=$HOME/osc_apps/git/2.9.0 CC=icc CFLAGS=-static-intel

# Make and make install steps
make && make install

You can use several variables that the installation script template defines. For our git installtion, their values are as follows:

More information is available in ~support/share/install-script/README.md. 
Below is an example of a full install script for git:

#!/bin/bash

source /users/PZS0645/support/share/install-script/install-template.sh

# List of files that should be in the install directory
VERIFY_FILES="
bin/git
"
initialize git 2.9.0

obtain_src() {
   # Download your source code in this step
   wget -P $dldir https://github.com/git/git/archive/v2.9.0.tar.gz
}

setup_step() {
   # Any steps to process the files obtained in `obtain_src`
   #   before they are ready to be configured
   tar zxf $dldir/v2.9.0.tar.gz -C $builddir --strip=1
}

configure_step() {
   # Any steps required to configure the source code before it can be compiled
   cd $builddir
   make configure
   ./configure --prefix=$installdir
}

make_step() {
   # Any steps required to build/compile the software
   cd $builddir
   make all
}

make_install_step() {
    # Any steps required to install the compiled software
   cd $builddir
   make install -j4
}

generate_module_file() {
cat <<EOF >>$modfile
prepend_path("PATH", root .. "/bin")
prepend_path("LD_LIBRARY_PATH", root .. "/lib")
prepend_path("LIBRARY_PATH", root .. "/lib")
prepend_path("INCLUDE", root .. "/include")
prepend_path("CPATH", root .. "/include")
prepend_path("FPATH", root .. "/include")
prepend_path("PKG_CONFIG_PATH", root .. "/lib/pkgconfig")
prepend_path("MANPATH", root .. "/share/man")
EOF
}

whatis_message() {
   # Echo text that should be put into whatis message.
   echo -e "\t$pkgname - loads the $pkgname package\n"
}

help_message() {
   # Echo text that should be put in help message.
   echo -e "\tHelp message for $pkgname version $pkgversion\n"
}

do_install
finalize

Running the install script

After our installation script is filled out, we need to run it. The first time you run the install template script, you need to run the command: git config --global --add safe.directory ~support/share/install-script to trust the install-script directory. This only needs to be done for your first local install, as the configuration will remain set for future installs. Run the following commands to make your install script executable and run it:

cd $HOME/osc_apps/git/install-script
chmod +x install-osc.sh
./install-osc.sh

If the installation failed, you need to remove any files beginning with INSTALL_FAILED before you can make changes and run it again. 

Once the script ran successfully, you will need to add your modulefiles to the modulepath with the command: module use $HOME/osc_apps/lmodfiles. Then load the module with module load git/2.9.0. 

To make sure you have the correct module file loaded, type which git which should emit "~/local/git/2.9.0/bin/git" (NOTE: ~ is equivalent to $HOME). 

To make sure the software was installed correctly and that the module is working, type git --version which should emit "git version 2.9.0".

For more information about modules, be sure to read the webpage indicated at the end of module help. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and oschelp@osc.edu.