Charm++ parallel programming system

[Shad Kirmani <sxk5292 AT cse.psu.edu>]

Hello,

When I run charm++ built with net-linux-x86_64, using the command:
../charmrun +p8 ../ChaNGa graphplot3d.param
I get the following errors:

**************************************************
[sxk5292@cyberstar163 testgraphplot]$ ../charmrun +p8 ../ChaNGa graphplot3d.param
Permission denied.
Permission denied.
Permission denied.
Permission denied.
Charmrun> Error 1 returned from rsh (localhost:0)
[sxk5292@cyberstar163 testgraphplot]$ Permission denied.
Permission denied.
Permission denied.
Permission denied.
***************************************************

Instead, when I compile charm++ with mpi-linux-x86_64-mpicxx, the code runs just fine. My cluster has inifiniband interconnect network. Can somebody please explain what are the requirements/permissions on the system to run charm++ compiled with net-linux-x86_64.

Thanks,
Shad

On Mon, Mar 12, 2012 at 12:48 PM, Mei, Chao <chaomei2 AT illinois.edu> wrote:

You could build charm++ either 

./build net-linux-x86_64 smp ibverbs --with-production

or 

./build net-linux-x86_64 ibverbs --with-production

The former is the SMP mode of charm++, the latter is the nonSMP mode of charm++.

Regarding the usage of SMP mode, one has to set ppn appropriately as Abhishek mentioned in his email.

For example, you want to run the program on 2 physical nodes (16 cores in total),

then in SMP, the command line will be:

./charmrun +p14 ./BIN ++ppn 7 +pemap 1-7,+commap 0

Note that we set ppn to 7 instead of 8 because we want to dedicate a core to the communication thread. "+pemap 1-7,+commap 0" is the way to set cpu affinity of those PEs.

In comparison, in nonSMP mode, the command line will be:

./charmrun +p16 ./BIN +pemap 0-7

Here, there's no communication thread, so the total 16 cores are used for the whole program.

-Chao

---

From: shad.kirmani AT gmail.com [shad.kirmani AT gmail.com] on behalf of Shad Kirmani [sxk5292 AT cse.psu.edu]
Sent: Monday, March 12, 2012 11:35 AM
To: Mei, Chao

Subject: Re: [ppl] [charm] Building charm++ with SMP and on GNU/Linux x86_64

Hello Mei,

The network on the cluster(8-way SMP and GNU/Linux x86_64) is Mellanox QDR Infiniband. So, I should go ahead and build it with "./build net-linux-x86_64 smp ibverbs --with-production" or "./build net-linux-amd64 ibverbs -j8 -optimize"?

The other details about the cluster are:
http://www.ics.psu.edu/infrast/specs.html

Yes, the programs do run on charm. I am just looking for an optimal way to run charm.

Thanks,
Shad

On Fri, Mar 9, 2012 at 9:57 PM, Mei, Chao <chaomei2 AT illinois.edu> wrote:

Hi Shad,

What's the network of the cluster that you use to run your program? If it is infiniband? If it is, then it is better to build charm++ with "./build net-linux-x86_64 smp ibverbs --with-production" than the MPI one. If you could tell us more about the cluster you used, it would be more helpful for us to point out the best option for building charm.

Secondly, have you tried to run a simple charm program to see if it works on your cluster?

Regards,

Chao Mei

---

From: ppl-bounces AT cs.uiuc.edu [ppl-bounces AT cs.uiuc.edu] on behalf of Abhishek Gupta [gupta59 AT illinois.edu]
Sent: Friday, March 09, 2012 8:08 PM
To: Shad Kirmani
Cc: charm AT cs.uiuc.edu
Subject: Re: [ppl] [charm] Building charm++ with SMP and on GNU/Linux x86_64

Hi Shad,

Is there a particular reason why you are trying to use mpi-linux instead of net-linux. In general, I would recommend that you use net-linux since it has better performance compared to mpi-linux. You can use the following command for building:

./build charm++ net-linux-x86_64 smp

Also, please tell the command that you are using to run your program. For SMP version, you need to specify a
+ppn workerThreadPerNode  runtime argument . You should leave one core for communication thread.e.g for 16 core node, you can use +ppn 15 to specify 15 worker threads per node.

Thanks,

Abhishek

On Fri, Mar 9, 2012 at 7:48 PM, Shad Kirmani <sxk5292 AT cse.psu.edu> wrote:

Hello,

I am trying to build charm++ with SMP support. My machine specifications are:

[sxk5292@cyberstar84 test]$ uname -a

Linux cyberstar84.hpc.rcc.psu.edu 2.6.18-274.7.1.el5 #1 SMP Mon Oct 17 11:57:14 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Please also visit http://www.ics.psu.edu/infrast/specs.html for more detailed specifications.

To build charm++, I chose the version mpi-linux-x86_64 with smp option set. 

When I run my code I get the following output from charm++:

[sxk5292@cyberstar84 test]$ ./pbsall.sh 

Charm++> Running on MPI version: 2.1

Charm++> level of thread support used: MPI_THREAD_SINGLE (desired: MPI_THREAD_FUNNELED)

Charm++> Running on SMP mode, 1 worker threads per process

Charm++> The comm. thread both sends and receives messages

Converse/Charm++ Commit ID: v6.3.0-1293-g7f245d0

Warning> Randomization of stack pointer is turned on in kernel.

------------- Processor 14 Exiting: Caught Signal ------------

Signal: 11

------------- Processor 49 Exiting: Caught Signal ------------

Signal: 11

--------------------------------------------------------------------------

mpirun noticed that process rank 14 with PID 21394 on node cyberstar83 exited on signal 11 (Segmentation fault).

--------------------------------------------------------------------------

...

...

I have tried building charm++ with the following command line arguments:

./build charm++ mpi-linux-x86_64 smp 

and

./build charm++ mpi-linux-x86_64 smp -DCMK_SMP=1 -DCMK_MPI_INIT_THREAD=1

Can anybody please help with the command line arguments to build charm++.

Thanks,

Shad

_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm

_______________________________________________
ppl mailing list
ppl AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/ppl