Charm++ parallel programming system

[Jozsef Bakosi <jbakosi AT gmail.com>]

Hi folks,

I'm wondering what is the best way to run Charm++ applications on clusters with Infiniband interconnects. So far I've been successfully building and running my app using Charm++, built by the following command, which uses MPI:

./build AMPI mpi-linux-x86_64 mpicxx

But now I'm wondering if the "ibverbs" build option provides better performance on Infiniband clusters. We have Qlogic and Mellanox Infiniband Fat-Tree interconnets. To experiment with this, I have successfully built Charm++ using the following command:

./build AMPI net-linux-x86_64 ibverbs

But when I try to run net-linux-x86_64-ibverbs/tests/charm++/simplearrayhello on two compute nodes, I get

$ ./charmrun +p32 ./hello

Charmrun> IBVERBS version of charmrun

mcmd: connect failed: Connection refused (32x)

Charmrun> Error 1 returned from rsh (localhost:0)

So my questions are:

1. Can I expect better performance on Infiniband clusters using build options other than MPI?

2. Do I also have to contact our system admins to allow access to lower (than MPI) level software layers for the interconnect so Charm++ code (I assume ibverbs) can use it?

3. Am I missing something else?

4. Are the best ways to build Charm++ for specific hardware documented somewhere?

Thanks in advance, and please let me know if I you need more information on the clusters.

Jozsef