Python

Python is a high-level, multi-paradigm programming language that is both easy to learn and useful in a wide variety of applications.  Python has a large standard library as well as a large number of third-party extensions, most of which are completely free and open source. 

Availability and Restrictions

Versions

Python is available on Pitzer and Owens Clusters. The versions currently available at OSC are:

Version Owens Pitzer Ascend Notes
2.7 X      
3.5  X      
3.6 X      
2.7-conda5.2 X X   Anaconda 5.2 distribution with Python 2.7 (conda 4.5.9 on Owens, conda 4.5.10 on Pitzer)**
3.6-conda5.2 X* X*   Anaconda 5.2 distribution with Python 3.6 (conda 4.5.9 on Owens, conda 4.5.11 on Pitzer)**

3.7-2019.10

X X   Anaconda 2019.10 distribution with Python 3.7 (conda 4.7.12)**
3.9-2022.05 X X   Anaconda 2022.05 distribution with Python 3.9 (conda 4.12.0)**
3.9     X*  
* Current default version
** The sufix '-condaX.X' and '-20XX.XX' indicates the version of the Anaconda distribution that has been installed. These distributions encompass conda as well as various other packages. For example, python/2.7-conda5.2 has been installed with Anaconda version 5.2 but uses conda version 4.5

Some versions installed as an integrated package Anaconda

You can use module spider python to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

Python is available for use by all OSC users.

Publisher/Vendor/Repository and License Type

Python Software Foundation, Open source

Usage

Terminal

Set-up

To load the default version of Python module, use module load python . To select a particular software version, use module load python/version. For example, use module load python/3.5 to load Python version 3.5. After the module is loaded, you can run the interpreter by using the command python. To unload the Python 3.5 module, use the command module unload python/3.5 or simply module unload python

Installed Modules

We have installed a number of Python packages and tuned them for optimal performance on our systems.  When using the Anaconda distributions of python you can run conda list to view the installed packages.

NOTE:
  • Due to architecture differences between our supercomputers, we recommend NOT installing your own packages in  ~/.local. Instead, you should install them in some other directory and set $PYTHONPATH in your default environment. For more information about installing your own Python modules, please see our HOWTO.
Environments

See the HOWTO section for more information on how to create and use python environements.

Batch

When you log into owens.osc.edu or pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Here is an example batch job script

#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1 
#SBATCH --time=00:01:00

module load python/3.9-2022.05

cp example.py $TMPDIR
cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR
    

Utilizing Python Environments Within Batch Job:

Important: When utilizing a python environment make sure to deactivate the environment before submitting the script or include  source deactivate in the batch script before activating the environment.
Here is an example batch job script involving conda environment:
#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1
#SBATCH --time=00:01:00

# run to following to ensure local environment does not effect the batch job in unexpected ways

source deactivate # deactivate copy of local python environment if job submitted from within environment
module reset      # reset any loaded modules

module load python/3.9-2022.05 # load python
export PYTHONNOUSERSITE=True  #to avoid local python packages

source activate MY_ENV  # activate conda environment 


# Rest of script below

cp example.py $TMPDIR

cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR

Jupyter

Launching Jupyter App

Log on to https://ondemand.osc.edu/ with your OSC credentials. Choose Jupyter under the InteractiveApps option. ondemand_home.jpeg

 

 

Provide job submission parameters then click Launch.

jupyter_settings.png

 

 

The next page shows the status of your job either as Queued or Starting or Running. Your job may sit in a queue for a few minutes depending on cluster load and resources requested.

jupyter_queued.png

 

 

When the job is ready, please click on Connect to Jupyter. This will now launch a Jupyter App.

jupyter_running.png

Jupyter App Usage 

With the app open, you will be able to access your home directory on the left and all your available kernels will appear on the right. Any custom kernels created using HOWTO: create virtual environment with jupyter will also appear in this selection.

Jupyter_main_menu.png

 

With a file open you can easily switch between different kernels by clicking the kernel name in the top right.

Kernel_swap.jpeg

HOW-TOs

Manage your Python packages

We highly recommend creating a local environment to manage Python packages for your production and research tasks. Please refer to the following how-to pages for more details:

Install packages for deep/machine learning

Advanced topics

 

Known Issues

Incorrect MPI launcher and compiler wrappers with Conda environments

Updated: March 2020
Versions Affected: Python 2.7, 3.6 & Conda 5.2
Users may encounter under-performing MPI jobs or failures of compiling MPI applications if you are using Conda from system. We found pre-installed mpich2 package in some Conda environments overrides default MPI path. The affected Conda packages are python/2.7-conda5.2 and python/3.6-conda5.2. If users experience these issues, please re-load MPI module, e.g. module load mvapich2 after setting up your Conda environment.

Further reading

Extensive documentation of the Python programming language and software downloads can be found at the Official Python Website.  

See Also

Supercomputer: 
Service: 
Fields of Science: