Set-up on Owens

We recommend to run STAR-CCM+ on only the compute nodes. Thus, all STAR-CCM+ jobs should be submitted via the batch scheduling system, either as interactive or non-interactive batch jobs. To load the default version of STAR-CCM+ module on Owens, use  module load starccm . To select a particular software version, use   module load starccm/version . For example, use  module load starccm/11.02.010  to load STAR-CCM+ version 11.02.010 on Owens.

Batch Usage on Owens

When you log into owens.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your STAR-CCM+ analysis to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used. STAR-CCM+ can be run on OSC clusters in either interactive mode or in non-interactive batch mode.

Interactive Batch Session

Interactive mode is similar to running STAR-CCM+ on a desktop machine in that the graphical user interface (GUI) will be sent from OSC and displayed on the local machine. To run interactive STAR-CCM+, it is suggested to request necessary compute resources from the login node, with X11 forwarding. The intention is that users can run STAR-CCM+ interactively for the purpose of building their model, preparing input file (.sim file), and checking results. Once developed this input file can then be run in no-interactive batch mode. For example, the following line requests one node with 28 cores( -N 1 -n 28 ), for a walltime of one hour ( -t 1:00:00 ), with one STAR-CCM+ base license token ( -L starccm@osc:1 ) on Owens:

sinteractive -N 1 -n 28 -t 1:00:00 -L starccm@osc:1

This job will queue until resources become available. Once the job is started, you're automatically logged in on the compute node; and you can launch STAR-CCM+ GUI with the following commands:

module load starccm
starccm+ -mesa

Non-interactive Batch Job (Serial Run using 1 Base Token)

A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice.

Below is the example batch script ( job.txt ) for a serial run with an input file ( starccm.sim ) on Owens:

#!/bin/bash
#SBATCH --job-name=starccm_test
#SBATCH --time=1:00:00
#SBATCH --nodes=1 --ntasks-per-node=1
#SBATCH -L starccm@osc:1

cd $TMPDIR  
cp $SLURM_SUBMIT_DIR/starccm.sim .  
module load starccm  
starccm+ -batch starccm.sim >&output.txt  
cp output.txt $SLURM_SUBMIT_DIR

To run this job on OSC batch system, the above script is to be submitted with the command:

sbatch job.txt

Non-interactive Batch Job (Parallel Run using HPC Tokens)

To take advantage of the powerful compute resources at OSC, you may choose to run distributed STAR-CCM+ for large problems. Multiple nodes and cores can be requested to accelerate the solution time. The following shows an example script if you need 2 nodes with 28 cores per node on Owens using the inputfile named   starccm.sim   :

#!/bin/bash
#SBATCH --job-name=starccm_test
#SBATCH --time=3:00:00
#SBATCH --nodes=2 --ntasks-per-node=28
#SBATCH -L starccm@osc:1,starccmpar@osc:55

cp starccm.sim $TMPDIR
cd $TMPDIR
module load starccm 

srun hostname | sort -n > ${SLURM_JOB_ID}.nodelist

starccm+ -np 56 -batch -machinefile ${SLURM_JOB_ID}.nodelist -mpi openmpi starccm.sim >&output.txt 
cp output.txt $SLURM_SUBMIT_DIR

In addition to requesting the STAR-CCM+ base license token ( -L starccm@osc:1 ), you need to request copies of the  starccmpar  license, i.e., HPC tokens ( -L starccm@osc:1,starccmpar@osc:[n] ), where [n] is equal to the number of cores minus 1.

We recommand using openmpi for your parallel jobs. Especially, 17.02.007 version would not work with intelmpi. 

starccm+ -np $SLURM_NTASKS -batch -machinefile ${SLURM_JOB_ID}.nodelist -mpi openmpi {your-input-file}

...
module load starccm/17.02.007
module load openmpi/4.0.3-hpcx
export OPENMPI_DIR=/usr/local/openmpi/intel/19.0/4.0.3-hpcx
srun hostname | sort -n > ${SLURM_JOB_ID}.nodelist
...
starccm+ -np $SLURM_NTASKS -batch -machinefile ${SLURM_JOB_ID}.nodelist -mpi openmpi {your-input-file}...
...