HOWTO: Add python packages using the conda package manager

While our python installations come with many popular packages installed, you may come upon a case where you need an addiditonal package that is not installed.  If the specific package you are looking for is available from Anaconda.org (formerlly binstar.org) you can easily install it and required dependencies by using the Conda package manager.

To be able to install a package using the conda package manager:

  • You must use a Anaconda distribution of python:
    • On Oakley the following modules:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Ruby:
      • python/2.7.latest*, python/2.7.8, python/3.4.2
    • On Owens:
      • python/2.7, python/3.5
  • * = latest suffix refers to a distribution that will be electively updated
  • Package should be available through Anaconda.org and/or using pip (see next section).

If you would like to freeze a distribution so that you can control how often you update Anaconda, please send us a help request at oschelp@osc.edu.

But I use virtualenv and/or pip!

See the comparison to these package management tools here:

https://conda.io/docs/commands.html#conda-vs-pip-vs-virtualenv-commands

Pip installations are supported:

module load python
module list # check which python you just loaded
pip install --upgrade --user PACKAGE # where PACKAGE is a valid package name

If binaries exist in a pip installation they are usually installed in:

$HOME/.local/bin

The one issue with this approach is portability with multiple Python modules.  If you plan to stick with a single Python module, then this should not be an issue.  However, if you commonly switch between Python 2 and Python 3, then be aware of the potential trouble in using the same installation location for all Python versions.

Virtualenv may also work, but is not under active testing at OSC.

Procedure

We will install the yt package to a local directory in this example.

Load proper python module

module load python/2.7.8

Clone python installation to local directory

conda create -n local --clone="$PYTHON_HOME"

This will clone the entire python installation to ~/envs/local directory. The process will take serveral minutes.

conda create -n local

This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.

conda create -n local python={version} anaconda

If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=2.4" for version 2.4.

Activate clone environment

source activate local

On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:

conda remove conda-build
conda remove conda-env

Install package

conda install yt
  • Replace yt with the name of the package you want to install, as listed by anaconda.org.
If there are errors on this step you will need to resolve them before continuing.

Test python package

Now we will test our installed python package by loading it in python and checking its location to ensure we are using the correct version.

python -c "import yt;print yt.__file__"

Output:

$HOME/.conda/envs/local/lib/python2.7/site-packages/yt/__init__.py
  • Replace both instances of yt  with the name of the package you installed.
Remember, you will need to load the proper version of python before you go to use your newlly installed package.  Packages are only installed to one version of python.

Install your own python modules

If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.

Further Reading:

Conda Test Drive: https://conda.io/docs/test-drive.html 

HOWTO: Install your own python modules

While we provide a number of Python modules, you may need a module we do not provide. If it is a commonly used module, or one that is particularly difficult to compile, you can contact OSC Help for assistance, but we have provided an example below showing how to build and install your own Python modules, and make them available inside of Python. Note, these instructions use "bash" shell syntax; this is our default shell, but if you are using something else (csh, tcsh, etc), some of the syntax may be different.

Please consider using conda python package manager before you try to build python using the method explained here. We have an instruction on conda here.

Gather your materials

First, you need to collect up what you need in order to do the installation. To keep things tidy, we will do all of our work in $HOME/local/src . You should make this directory now.

mkdir -p $HOME/local/src

Now, we will need to download the source code for the module we want to install. In our example, we will use "NumExpr", a module we already provide in the system version of Python. You can either download the file to your desktop, and then upload it to OSC, or directly download it using the wget utility (if you know the URL for the file).

cd ~/local/src
wget http://numexpr.googlecode.com/files/numexpr-2.0.1.tar.gz

Now, extract the downloaded file. In this case, since it's a "tar.gz" format, we can use tar to decompress and extract the contents.

tar xvfz numexpr-2.0.1.tar.gz

You can delete the downloaded archive now, if you wish, or leave it around should you want to start the installation from scratch.

Build it!

Environment

To build the module, we will want to first create a temporary environment variable to aid in installation. We'll call it "INSTALL_DIR".

export INSTALL_DIR=${HOME}/local/numexpr/2.0.1

I am following, roughly, the convention we use at the system level. This allows us to easily install new versions of software without risking breaking anything that uses older versions. We have specified a folder for the program (numexpr), and for the version (2.0.1). Now, to be consistent with python installations, we're going to create a second temporary environment variable, which will contain the actual installation location.

export TREE=${INSTALL_DIR}/lib/python2.7/site-packages

Now, make the directory tree.

mkdir -p $TREE

Compile

To compile the module, we should switch to the GNU compilers. The system installation of Python was compiled with the GNU compilers, and this will help avoid any unnecessary complications. We will also load the Python module, if it hasn't already been loaded.

module swap intel gnu
module load python

Now, build it. This step may vary a bit, depending on the module you are compiling. You can execute python setup.py --help to see what options are available. Since we are overriding the install path to one that we can write to, and that fits our management plan, we need to use the --prefix option.

python setup.py install --prefix=$INSTALL_DIR

Make it usable

At this point, the module is compiled and installed in ~/local/numexpr/2.0.1/lib/python2.7/site-packages . Occasionally, some files will be installed in ~/local/numexpr/2.0.1/bin as well. To ensure Python can locate these files, we need to modify our environment.

Manual

The most immediate way - but the one that must be repeated every time you wish to use the module - is to manually modify your environment. If files are installed in the "bin" directory, you'll need to add it to your path. As before, these examples are for bash, and may have to be modified for other shells. Also, you will have to modify the directories to match your install location.

export PATH=$PATH:~/local/numexpr/2.0.1/bin

And, for the python libraries:

export PYTHONPATH=$PYTHONPATH:~/local/numexpr/2.0.1/lib/python2.7/site-packages

Hardcode it

We don't really recommend this option, as it is less flexible, and can cause conflicts with system software. But, if you want, you can modify your .bashrc (or similar file, depending on your shell) to set these environment variables automatically. Be extra careful; making a mistake in .bashrc (or similar) can destroy your login environment in a way that will require a system administrator to fix. To do this, you can copy the lines above modifying $PATH and $PYTHONPATH into .bashrc. Remember - test them interactively first! If you destroy your shell interactively, the fix is as simple as logging out and then logging back in. If you break your login environment, you'll have to get our help to fix it.

Make a module (recommended!)

This is the most complicated option, but it is also the most flexible, as you can have multiple versions of this particular software installed, and specify at run-time which one to use. This is incredibly useful if a major feature changes that would break old code, for example. You can see our tutorial on writing modules here, but the important variables to modify are, again, $PATH and $PYTHONPATH . You should specify the complete path to your home directory here, and not rely on any shortcuts like ~ or $HOME .  Below is a modulefile written in Lua:

If you are following the tutorial on writing modules, you will want to place this file in $HOME/local/share/modulefiles/numexpr/2.0.1.lua :

-- This is a Lua modulefile, this file 2.0.1.lua can be located anywhere
-- But if you are following a local modulefile location convention, we place them in
-- $HOME/local/share/modulefiles/
-- For numexpr we place it in $HOME/local/share/modulefiles/numexpr/2.0.1.lua
-- This finds your home directory
local homedir = os.getenv("HOME")
prepend_path("PYTHONPATH", 
pathJoin(homedir, "/local/numexpr/2.0.1/lib/python2.7/site-packages"))
prepend_path(homedir, "local/numexpr/2.0.1/bin"))

 

Once your module is created (again, see the guide), you can use your python module simply by loaded the software module you created.

module use $HOME/local/share/modulefiles/
module load numexpr/2.0.1
Service: