While our Python installations come with many popular packages installed, you may come upon a case in which you need an additional package that is not installed. If the specific package you are looking for is available from anaconda.org (formerlly binstar.org), you can easily install it and required dependencies by using the conda package manager.
The following steps are an example of how to set up a Python environment and install packages to a local directory using conda. We use the name local for the environment, but you may use any other name.
We have python and miniconda3 modules. python modules are based on Anaconda package manager, and miniconda3 module is based on Miniconda package manager. python modules are typically recommended when you use Python in a standard environment that we provide. However, if you want to create your own python environment, we recommend using miniconda3 module, since you can start with minimal configurations.
module load miniconda3
Three alternative create commands are listed. These cover the most common cases.
The following will create a minimal Python installation without any extraneous packages:
conda create -n local
If you want to clone the full base Python environment from the system, you may use the following create command:
conda create -n local --clone base
You can augment the command above by listing specific packages you would like installed into the environment. For example, the following will create a minimal Python installation with only the specified packages (in this case, numpy and babel):
conda create -n local numpy babel
By default, conda will install the newest versions of the packages it can find. Specific versions can be specified by adding =<version> after the package name. For example, the following will create a Python installation with Python version 2.7 and NumPy version 1.16:
conda create -n local python=2.7 numpy=1.16
To verify that a clone has been created, use the command
conda info -e
For additional conda command documentation see https://docs.conda.io/projects/conda/en/latest/commands.html#conda-general-commands
Before the created environment can be used, it must be activated.
For the bash shell:
source activate local
conda create step, you may saw a message from the installer that you can use conda activate command for activating environment. But, please don't use conda activate command, because it will try to update your shell configuration file and it may cause other issues. So, please use source activate command as we suggest above.On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:
conda remove conda-build
conda remove conda-env
To install additional packages, use the conda install command. For example, to install the yt package:
conda install yt
By default, conda will install the newest version if the package that it can find. Specific versions can be specified by adding =<version> after the package name. For example, to install version 1.16 of the NumPy package:
conda install numpy=1.16
If you need to install packages with pip, then you can install pip in your virtual environment by
conda install pip
Then, you can install packages with pip as
pip install PACKAGE
export PYTHONNOUSERSITE=True to prevent packages from user's .local path.Now we will test our installed Python package by loading it in Python and checking its location to ensure we are using the correct version. For example, to test that NumPy is installed correctly, run
python -c "from __future__ import print_function; import numpy; print(numpy.__file__)"
and verify that the output generally matches
$HOME/.conda/envs/local/lib/python3.6/site-packages/numpy/__init__.py
To test installations of other packages, replace all instances of numpy with the name of the package you installed.
If the method using conda above is not working, or if you prefer, you can consider installing Python packages from the source. Please read HOWTO: install your own Python packages.
See the comparison to these package management tools here:
https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands
pip installations are supported:
module load python module list # check which python you just loaded pip install --user --upgrade PACKAGE # where PACKAGE is a valid package name
Note the default installation prefix is set to the system path where OSC users cannot install the package. With the option --user, the prefix is set to $HOME/.local where lib, bin, and other top-level folders for the installed packages are placed. Finally, the option --upgrade will upgrade the existing packages to the newest available version.
The one issue with this approach is portability with multiple Python modules. If you plan to stick with a single Python module, then this should not be an issue. However, if you commonly switch between different Python versions, then be aware of the potential trouble in using the same installation location for all Python versions.
Typically, you can install packages with the methods shown in Install packages section above, but in some cases where the conda package installations have no source from conda channels or have dependency issues, you may consider using pip in an isolated Python virtual environment.
To create an isolated virtual environment:
module reset python3 -m venv --without-pip $HOME/venv/mytest --prompt "local" source $HOME/venv/mytest/bin/activate (local) curl https://bootstrap.pypa.io/get-pip.py |python # get the newest version of pip (local) deactivate
where we use the path $HOME/venv/mytest and the name local for the environment, but you may use any other path and name.
To activate and deactivate the virtual environment:
source $HOME/venv/mytest/bin/activate (local) deactivate
To install packages:
source $HOME/venv/mytest/bin/activate (local) pip install PACKAGE
You don't need the --user option within the virtual environment.
Conda Test Drive: https://conda.io/docs/test-drive.html
This documentation describes how to install tensorflow package locally in your $HOME space.
Load python module
module load python/3.6-conda5.2
Three alternative create commands are listed. These cover the most common cases:
conda create -n local --clone="$PYTHON_HOME"
This will clone the entire python installation to ~/envs/local directory. The process will take several minutes.
conda create -n local
This will create a local python installation without any packages. If you need a small number of packages, you may choose this option.
conda create -n local python={version} anaconda
If you like to install a specific version of python, you can specify it with "python" option. For example, you can use "python=3.6" for version 3.6.
To verify that a clone has been created, use the command
conda info -e
For additional conda command documentation see https://conda.io/docs/commands.html
For the bash shell:
source activate local
On newer versions of Anaconda on the Owens cluster you may also need to perform the removal of the following packages before trying to install your specific packages:
conda remove conda-build
conda remove conda-env
Install the latest version of tensorflow that is gpu compatible.
pip install tensorflow-gpu
Now we will test tensorflow package by loading it in python and checking its location to ensure we are using the correct version.
python -c "import tensorflow;print (tensorflow.__file__)"
Output:
$HOME/.conda/envs/local/lib/python2.7/site-packages/tensorflow/__init__.py
If the method using conda above is not working or if you prefer, you can consider installing python modules from the source. Please read HOWTO: install your own python modules.
While we provide a number of Python packages, you may need a package we do not provide. If it is a commonly used package or one that is particularly difficult to compile, you can contact OSC Help for assistance. We also have provided an example below showing how to build and install your own Python packages and make them available inside of Python. These instructions use "bash" shell syntax, which is our default shell. If you are using something else (csh, tcsh, etc), some of the syntax may be different.
First, you need to collect what you need in order to perform the installation. We will do all of our work in $HOME/local/src. You should make this directory now.
mkdir -p $HOME/local/src
Next, we will need to download the source code for the package we want to install. In our example, we will use "NumExpr," a package we already provide in the system version of Python. You can either download the file to your desktop and then upload it to OSC, or directly download it using the wget utility (if you know the URL for the file).
cd ~/local/src wget http://numexpr.googlecode.com/files/numexpr-2.0.1.tar.gz
Next, extract the downloaded file. In this case, since it's a "tar.gz" format, we can use tar to decompress and extract the contents.
tar xvfz numexpr-2.0.1.tar.gz
You can delete the downloaded archive now or keep it should you want to start the installation from scratch.
To build the package, we will want to first create a temporary environment variable to aid in installation. We'll call INSTALL_DIR.
export INSTALL_DIR=${HOME}/local/numexpr/2.0.1
We are roughly following the convention we use at the system level. This allows us to easily install new versions of software without risking breaking anything that uses older versions. We have specified a folder for the program (numexpr), and for the version (2.0.1). To be consistent with Python installations, we will create a second temporary environment variable that will contain the actual installation location.
export TREE=${INSTALL_DIR}/lib/python2.7/site-packages
Next, make the directory tree.
mkdir -p $TREE
To compile the package, we should switch to the GNU compilers. The system installation of Python was compiled with the GNU compilers, and this will help avoid any unnecessary complications. We will also load the Python package, if it hasn't already been loaded.
module swap intel gnu module load python
Next, build it. This step may vary a bit, depending on the package you are compiling. You can execute python setup.py --help to see what options are available. Since we are overriding the install path to one that we can write to and that fits our management plan, we need to use the --prefix option.
python setup.py install --prefix=$INSTALL_DIR
At this point, the package is compiled and installed in ~/local/numexpr/2.0.1/lib/python2.7/site-packages. Occasionally, some files will be installed in ~/local/numexpr/2.0.1/bin as well. To ensure Python can locate these files, we need to modify our environment.
The most immediate way -- but the one that must be repeated every time you wish to use the package -- is to manually modify your environment. If files are installed in the "bin" directory, you'll need to add it to your path. As before, these examples are for bash, and may have to be modified for other shells. Also, you will have to modify the directories to match your install location.
export PATH=$PATH:~/local/numexpr/2.0.1/bin
And for the Python libraries:
export PYTHONPATH=$PYTHONPATH:~/local/numexpr/2.0.1/lib/python2.7/site-packages
We don't recommend this option, as it is less flexible and can cause conflicts with system software. But if you want, you can modify your .bashrc (or similar file, depending on your shell) to set these environment variables automatically. Be extra careful; making a mistake in .bashrc (or similar) can destroy your login environment in a way that will require a system administrator to fix. To do this, you can copy the lines above modifying $PATH and $PYTHONPATH into .bashrc. Remember to test them interactively first. If you destroy your shell interactively, the fix is as simple as logging out and then logging back in. If you break your login environment, you'll have to get our help to fix it.
This is the most complicated option, but it is also the most flexible, as you can have multiple versions of this particular software installed and specify at run-time which one to use. This is incredibly useful if a major feature changes that would break old code, for example. You can see our tutorial on writing modules here, but the important variables to modify are, again, $PATH and $PYTHONPATH. You should specify the complete path to your home directory here and not rely on any shortcuts like ~ or $HOME. Below is a modulefile written in Lua:
If you are following the tutorial on writing modules, you will want to place this file in $HOME/local/share/modulefiles/numexpr/2.0.1.lua:
-- This is a Lua modulefile, this file 2.0.1.lua can be located anywhere
-- But if you are following a local modulefile location convention, we place them in
-- $HOME/local/share/modulefiles/
-- For numexpr we place it in $HOME/local/share/modulefiles/numexpr/2.0.1.lua
-- This finds your home directory
local homedir = os.getenv("HOME")
prepend_path("PYTHONPATH",
pathJoin(homedir, "/local/numexpr/2.0.1/lib/python2.7/site-packages"))
prepend_path(homedir, "local/numexpr/2.0.1/bin"))
Once your module is created (again, see the guide), you can use your Python package simply by loading the software module you created.
module use $HOME/local/share/modulefiles/ module load numexpr/2.0.1