Charm++ parallel programming system

[Abhinav Bhatele <bhatele AT illinoisalumni.org>]

Hi Jozsef,

Please find some answers inline:

On Fri, Feb 13, 2015 at 8:19 AM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:

Hi folks,

I'm wondering what is the best way to run Charm++ applications on clusters with Infiniband interconnects. So far I've been successfully building and running my app using Charm++, built by the following command, which uses MPI:

./build AMPI mpi-linux-x86_64 mpicxx

But now I'm wondering if the "ibverbs" build option provides better performance on Infiniband clusters. We have Qlogic and Mellanox Infiniband Fat-Tree interconnets. To experiment with this, I have successfully built Charm++ using the following command:

./build AMPI net-linux-x86_64 ibverbs

But when I try to run net-linux-x86_64-ibverbs/tests/charm++/simplearrayhello on two compute nodes, I get

$ ./charmrun +p32 ./hello

Charmrun> IBVERBS version of charmrun

mcmd: connect failed: Connection refused (32x)

Charmrun> Error 1 returned from rsh (localhost:0)

So my questions are:

1. Can I expect better performance on Infiniband clusters using build options other than MPI?

Yes, typically you would expect the ibverbs build to perform better than the MPI build. You can try the four builds below:

mpi-linux-x86_64 mpicxx

mpi-linux-x86_64 mpicxx smp

net-linux-x86_64 ibverbs

net-linux-x86_64 ibverbs smp

 

2. Do I also have to contact our system admins to allow access to lower (than MPI) level software layers for the interconnect so Charm++ code (I assume ibverbs) can use it?

No, like Jim pointed out you can use ++mpiexec or manually specify the nodelist that has been allocated to you:

http://charm.cs.illinois.edu/manuals/html/charm++/C.html

 

3. Am I missing something else?

4. Are the best ways to build Charm++ for specific hardware documented somewhere?

Hopefully, someone else will answer this but my guess is no.

 

Thanks in advance, and please let me know if I you need more information on the clusters.

Jozsef

_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm

_______________________________________________
ppl mailing list
ppl AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/ppl

--

Abhinav Bhatele, people.llnl.gov/bhatele
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory