Charm++ parallel programming system

["Sangamesh B" <forum.san AT gmail.com>]

Hi,

   I'm building charm5.9 on Rocks 4.3 cluster with intel 10 compilers and
mvapich2-1.0.3 infiniband support.

  The charm build is successful with following options:

./build charm++ mpi-linux iccstatic pthreads --no-build-shared
--incdir=/opt/MPI/mvapich2_intel/include
--libdir=/opt/MPI/mvapich2_intel/lib --libdir=/usr/lib

 But when I try to execute tests/charm++/simplearrayhello program:

[root@local
 simplearrayhello]# make
../../../bin/charmc   hello.ci
../../../bin/charmc  -c hello.C
../../../bin/charmc  -language charm++ -o hello hello.o
icpc: command line remark #10010: option '-static-libcxa' is deprecated and
will be removed in a future release. See '-help deprecated'
ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthre

..
..
ld: skipping incompatible /usr/lib/libc.so when searching for -lc
ld: skipping incompatible /usr/lib/libc.a when searching for -lc
/opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0xa7): In
function `MPIDI_CH3I_CM_Finalize':
: undefined reference to `ibv_dereg_mr'
/opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0xb9): In
function `MPIDI_CH3I_CM_Finalize':
: undefined reference to `ibv_dereg_mr'
/opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0x112): In
function `MPIDI_CH3I_CM_Finalize':
: undefined reference to `ibv_destroy_qp'


...
Fatal Error by charmc in directory
/opt/apps/namd_intel/charm-5.9/tests/charm++/simplearrayhello
   Command icpc -static-libcxa -L/opt/MPI/mvapich2_intel/lib -L/usr/lib
-L/opt/MPI/mvapich2_intel/lib -L/usr/lib -o hello -L../../../bin/../lib
-I../../../bin/../include ../../../bin/../lib/libldb-rand.o hello.o
moduleinit3173.o ../../../bin/../lib/libmemory-default.o
../../../bin/../lib/libthreads-default.o -lck -lconv-cplus-y -lconv-core
-lconv-util -L/opt/MPI/mvapich2_intel/lib -L/usr/lib -lckqt -lmpich
-lpthread -lpthread -ldl -lpthread -lm returned error code 1
charmc exiting...
make: *** [hello] Error 1

It looks, charmc is not able to link libibverbs.

What is the solution for this?

The src/arch/mpi-linux/conv-mach.sh is:

CMK_CPP_CHARM='/lib/cpp -P'
CMK_CPP_C='mpicc -E'
CMK_CC='mpicc '
CMK_CXX='mpicxx '
CMK_CXXPP='mpicxx -E '
CMK_CF77='ifort'
CMK_CF90='ifort'
CMK_RANLIB='ranlib'
CMK_LIBS='-L/opt/MPI/mvapich2_intel/lib -L/usr/lib -lckqt -lmpich -lpthread'
CMK_LD_LIBRARY_PATH="-Wl,-rpath,$CHARMLIBSO/"
CMK_NATIVE_LIBS=''
CMK_NATIVE_CC='icc '
CMK_NATIVE_LD='icpc'
CMK_NATIVE_CXX='icpc '
CMK_NATIVE_LDXX='icpc'
CMK_NATIVE_CC='icc '
CMK_NATIVE_CXX='icpc '
CMK_F90LIBS='-L/usr/absoft/lib -L/opt/absoft/lib -lf90math -lfio -lU77
-lf77math '
CMK_MOD_NAME_ALLCAPS=1
CMK_MOD_EXT="mod"
CMK_F90_USE_MODDIR=1
CMK_F90_MODINC="-p

Is this setting is ok or I need to add some more library paths?

The mvapich2 is built with  ofa stack.

Thank you,
Sangamesh