TotalView Debugger

TotalView is a symbolic debugger which supports threads, MPI, OpenMP, C/C++, CUDA and Fortran, plus mixed-language codes.  Advanced features include on-demand memory leak detection, heap allocation debugging and the Standard Template Library Viewer (STLView).  Other features such as dive (which quickly shows more detail), a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis and control at the thread level enable debugging of parallel programs.  

Availability and Restrictions

The versions currently available at OSC are:

Version Oakley Ruby Owens notes
8.9.2-1 X(CS)     Default version on Oakley prior to 09/15/2015
8.14.1-8 X X    
2016.04.08     X(S)  
2017.0.12 X(S)* X(CS)* X(CS)* New user Interface options available
2017.2.11     X(CS)  
* Current default version

NOTE:

  • X(S): standard version
  • X(CS): Both standard and CUDA versions are available
  • X: CUDA debugging included in standard module

You can use module spider totalview to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access and Licensing

TotalView is available for use by all OSC users.  The installed license permits the use of 32 tokens (processes).  This means that the maximum parallel tasks for a TotalView session is 32.  If the token limit is exceeded an error with the text "Unable to acquire a license for X processes." will be presented in a message box.  If you need to debug a program with more than 32 processes you can attach to a subset of the processes.

Usage

General Usage Notes

TotalView provides a graphical user interface (GUI) as well as a textual user interface. (In general, the command totalview, which invokes the GUI, can be swapped with the command totalviewcli, which invokes the Command-Line Interface.) For the GUI application, one should use an OnDemand VDI (Virtual Desktop Interface) or have X11 forwarding enabled (see Setting up X Windows). Note that X11 forwarding can be distractingly slow for interactive applications.

Debugging is typically done in an interactive batch environment.  This is a requirement for MPI programs and is also important for multithreaded programs.  See our documentation on Interactive Batch jobs.

Note:  Debugging of CUDA 8.0 programs requires at least version 2017 of TotalView.

Usage on Oakley

Building your code for debugging with TotalView

You should build your code with the -g option for symbolic debugging.  For example:

icc -g -o mycode mycode.c

or

mpicc -g -o mympicode mympicode.c

Set-up

To load the default standard version of Totalview module, use  module load totalview . To select a particular software version, use   module load totalview/version . For example, use  module load totalview/8.9.2-1 to load Totalview version 8.9.2-1. 

Using Totalview

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands. The -a argument is used only if you want to pass arguments to your program on the command line.

module load totalview
totalview mycode -a prog_args

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXT in this directory for instructions for using these examples.

To debug an MPI program, use the following commands in an interactive job:

module load totalview
mpiexec -tv [other mpiexec arguments] mympiprog [program arguments]

Totalview will stop your program at the beginning to allow you to debug. For a parallel program, the initial breakpoint is in mpiexec. Press "Go" and you will be given the option of stopping when your job goes parallel. Normally it will stop inside MPI_Init. For a parallel job running on a single node, however, it can run past that point and stop later in your program. You may need to have your program sleep for 30 seconds at the beginning.

In C:

sleep(30);

In Fortran:

call sleep(30)

Usage on Ruby

Building your code for debugging with TotalView

You should build your code with the -g  option for symbolic debugging.  For example:

icc -g -o mycode mycode.c

or

mpicc -g -o mympicode mympicode.c

Set-up

To load the default standard version of Totalview module, use  module load totalview . To select a particular software version, use   module load totalview/version . For example, use  module load totalview/8.14.1-8 to load Totalview version 8.14.1-8. 

Using Totalview

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands. The -a argument is used only if you want to pass arguments to your program on the command line.

module load totalview
totalview ./mycode -a prog_args

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXT in this directory for instructions for using these examples.

Use the following commands in an interactive job:

module load totalview
totalview -args mpiexec [mpiexec arguments] ./mympiprog [program arguments]

Totalview will stop your program at the beginning to allow you to debug. For a parallel program, the initial breakpoint is in mpiexec. Press "Go" and you will be given the option of stopping when your job goes parallel. Normally it will stop inside MPI_Init. For a parallel job running on a single node, however, it can run past that point and stop later in your program. You may need to have your program sleep for 30 seconds at the beginning.

In C:

sleep(30);

In Fortran:

call sleep(30)

Usage on Owens

Building your code for debugging with TotalView

You should build your code with the -g  option for symbolic debugging.  For example:

icc -g -o mycode mycode.c

or

mpicc -g -o mympicode mympicode.c

Set-up

To load the default standard version of Totalview module, use  module load totalview . To select a particular software version, use   module load totalview/version . For example, use  module load totalview/2016.04.08 to load Totalview version 2016.04.08. 

Using Totalview

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands. The -a argument is used only if you want to pass arguments to your program on the command line.

module load totalview
totalview ./mycode -a prog_args

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXTin this directory for instructions for using these examples.

Use the following commands in an interactive job:

module load totalview
totalview -args mpiexec [mpiexec arguments] ./mympiprog [program arguments]

Totalview will stop your program at the beginning to allow you to debug. For a parallel program, the initial breakpoint is in mpiexec. Press "Go" and you will be given the option of stopping when your job goes parallel. Normally it will stop inside MPI_Init. For a parallel job running on a single node, however, it can run past that point and stop later in your program. You may need to have your program sleep for 30 seconds at the beginning.

In C:

sleep(30);

In Fortran:

call sleep(30)

Further Reading

  • On Oakley and Ruby, TotalView documentation in pdf format can be found in  $TOTALVIEW_HOME/doc/pdf
    • TotalView_for_HPC_Getting_Started_Guide.pdf
    • TotalView_for_HPC_User_Guide.pdf
    • Debugging_Memory_Problems_with_MemoryScape.pdf
  • Additional information about TotalView can be found at the Rogue Wave Software page.  
Service: 
Fields of Science: