Over the past two weeks we have experienced Oakely login node crashes potentially caused by a Lustre bug.

TotalView Debugger


TotalView is a symbolic debugger which supports threads, MPI, OpenMP, C/C++ and Fortran, plus mixed-language codes.  Advanced features include on-demand memory leak detection, heap allocation debugging and the Standard Template Library Viewer (STLView).  Other features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis and control at the thread level give you the power you need to solve tough problems.  


Version Glenn Oakley RUBY
8.3.0 X    
8.5.0-1 X    
8.8.0-2 X    
8.9.2-1   X*  
8.14.1-8   X** X**

* Both standard and CUDA versions are available.  

** CUDA debugging included in standard module.


Building your code for debugging with TotalView

You should build your code with the -g option for symbolic debugging.  For example:

icc -g -c mycode.c


mpicc -g -c mympicode.c

To debug an MPI program on Ruby, you must build with a debug version of mvapich2.  Swap modules before building code, for example (Ruby only):

module swap mvapich2/1.9 mvapich2/1.9-debug

Launching TotalView

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands:

module load totalview

Example code to debug can be found on Glenn, Oakley and Ruby in $TOTALVIEW_HOME/linux-x86-64/examples.  Refer to the file README.TXT in this directory for instructions for using these examples.

To debug an MPI program on Oakley or Glenn, use the following commands in an interactive job:

module load totalview
mpiexec -tv [other mpiexec arguments] mympiprog [program arguments]

Note: It may be necessary to use mpiexec.tv -tv on Oakley instead of just mpiexec -tv.

To debug an MPI program on Ruby, use the following commands in an interactive job:

module load totalview
totalview -args mpiexec [mpiexec arguments] ./mympiprog [program arguments]

Further Reading

On Oakley and Ruby, TotalView documentation in pdf format can be found in $TOTALVIEW_HOME/doc/pdf.  

Additional information about TotalView can be found at the Rogue Wave Software page.  

Fields of Science: