STAR-CCM+ encounters errors when running MPI jobs using Intel MPI or Open MPI, displaying the following message:
ib_iface.c:1139 UCX ERROR Invalid active_speed on mlx5_0:1: 128
This issue occurs because the UCX library (v1.8) bundled with STAR-CCM+ only supports Mellanox InfiniBand EDR, whereas the Cardinal cluster utilizes newer Mellanox InfiniBand NDR (Next Data Rate) fabric. Because UCX v1.8 cannot recognize the NDR speed, STAR-CCM+ fails to communicate across the network.
Affected versions
- Intel MPI: 18.06.006, 19.04.009, 19.06.009, and potentially later versions.
- Open MPI: 18.06.006 (specifically when running multi-node MPI jobs).
Workaround
To resolve this, you must bypass the UCX library for MPI communication by configuring the appropriate environment variables or command-line flags.
For Intel MPI, set the following environment variables in your script before executing the starccm+ command:
export -n I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
export I_MPI_HYDRA_BOOTSTRAP=ssh
export FI_PR=
For OpenMPI, append the following flags directly to your starccm+ execution command:
-mpiflags "--mca pml ^ucx --mca btl_openib_allow_ib 1"
