Charm++ parallel programming system

[Phil Miller <mille121 AT illinois.edu>]

The easiest thing to try would be adding ++mpiexec to your charmrun command:

 ../charmrun ++mpiexec +p8 ../ChaNGa graphplot3d.param

This will use the system's own process launcher to actually spawn the
job, but will otherwise operate as usual.

If that doesn't work, see what happens if you manually try to ssh to
one of the worker nodes in your cluster. Does that also give a
'Permission denied' error?

Phil

On Fri, Mar 23, 2012 at 15:05, Shad Kirmani 
<sxk5292 AT cse.psu.edu>
 wrote:
> Hello,
>
> When I run charm++ built with net-linux-x86_64, using the command:
> ../charmrun +p8 ../ChaNGa graphplot3d.param
> I get the following errors:
>
> **************************************************
> [sxk5292@cyberstar163
>  testgraphplot]$ ../charmrun +p8 ../ChaNGa
> graphplot3d.param
> Permission denied.
> Permission denied.
> Permission denied.
> Permission denied.
> Charmrun> Error 1 returned from rsh (localhost:0)
> [sxk5292@cyberstar163
>  testgraphplot]$ Permission denied.
> Permission denied.
> Permission denied.
> Permission denied.
> ***************************************************
>
> Instead, when I compile charm++ with mpi-linux-x86_64-mpicxx, the code runs
> just fine. My cluster has inifiniband interconnect network. Can somebody
> please explain what are the requirements/permissions on the system to run
> charm++ compiled with net-linux-x86_64.
>
> Thanks,
> Shad
>
>
> On Mon, Mar 12, 2012 at 12:48 PM, Mei, Chao 
> <chaomei2 AT illinois.edu>
>  wrote:
>>
>> You could build charm++ either
>>
>> ./build net-linux-x86_64 smp ibverbs --with-production
>>
>> or
>>
>> ./build net-linux-x86_64 ibverbs --with-production
>>
>> The former is the SMP mode of charm++, the latter is the nonSMP mode of
>> charm++.
>>
>> Regarding the usage of SMP mode, one has to set ppn appropriately
>> as Abhishek mentioned in his email.
>>
>> For example, you want to run the program on 2 physical nodes (16 cores in
>> total),
>> then in SMP, the command line will be:
>>
>> ./charmrun +p14 ./BIN ++ppn 7 +pemap 1-7,+commap 0
>>
>> Note that we set ppn to 7 instead of 8 because we want to dedicate a core
>> to the communication thread. "+pemap 1-7,+commap 0" is the way to set cpu
>> affinity of those PEs.
>>
>> In comparison, in nonSMP mode, the command line will be:
>> ./charmrun +p16 ./BIN +pemap 0-7
>> Here, there's no communication thread, so the total 16 cores are used for
>> the whole program.
>>
>> -Chao
>>
>> ________________________________
>> From: 
>> shad.kirmani AT gmail.com
>>  
>> [shad.kirmani AT gmail.com]
>>  on behalf of Shad
>> Kirmani 
>> [sxk5292 AT cse.psu.edu]
>> Sent: Monday, March 12, 2012 11:35 AM
>> To: Mei, Chao
>>
>> Subject: Re: [ppl] [charm] Building charm++ with SMP and on GNU/Linux
>> x86_64
>>
>> Hello Mei,
>>
>> The network on the cluster(8-way SMP and GNU/Linux x86_64) is Mellanox QDR
>> Infiniband. So, I should go ahead and build it with "./build
>> net-linux-x86_64 smp ibverbs --with-production" or "./build net-linux-amd64
>> ibverbs -j8 -optimize"?
>>
>> The other details about the cluster are:
>> http://www.ics.psu.edu/infrast/specs.html
>>
>> Yes, the programs do run on charm. I am just looking for an optimal way to
>> run charm.
>>
>> Thanks,
>> Shad
>>
>> On Fri, Mar 9, 2012 at 9:57 PM, Mei, Chao 
>> <chaomei2 AT illinois.edu>
>>  wrote:
>>>
>>> Hi Shad,
>>>
>>> What's the network of the cluster that you use to run your program? If it
>>> is infiniband? If it is, then it is better to build charm++ with "./build
>>> net-linux-x86_64 smp ibverbs --with-production" than the MPI one. If you
>>> could tell us more about the cluster you used, it would be more helpful 
>>> for
>>> us to point out the best option for building charm.
>>>
>>> Secondly, have you tried to run a simple charm program to see if it works
>>> on your cluster?
>>>
>>> Regards,
>>> Chao Mei
>>>
>>> ________________________________
>>> From: 
>>> ppl-bounces AT cs.uiuc.edu
>>>  
>>> [ppl-bounces AT cs.uiuc.edu]
>>>  on behalf of
>>> Abhishek Gupta 
>>> [gupta59 AT illinois.edu]
>>> Sent: Friday, March 09, 2012 8:08 PM
>>> To: Shad Kirmani
>>> Cc: 
>>> charm AT cs.uiuc.edu
>>> Subject: Re: [ppl] [charm] Building charm++ with SMP and on GNU/Linux
>>> x86_64
>>>
>>> Hi Shad,
>>>
>>> Is there a particular reason why you are trying to use mpi-linux instead
>>> of net-linux. In general, I would recommend that you use net-linux since 
>>> it
>>> has better performance compared to mpi-linux. You can use the following
>>> command for building:
>>>
>>> ./build charm++ net-linux-x86_64 smp
>>>
>>> Also, please tell the command that you are using to run your program. For
>>> SMP version, you need to specify a
>>> +ppn workerThreadPerNode  runtime argument . You should leave one core
>>> for communication thread.e.g for 16 core node, you can use +ppn 15 to
>>> specify 15 worker threads per node.
>>>
>>> Thanks,
>>>
>>> Abhishek
>>>
>>>
>>> On Fri, Mar 9, 2012 at 7:48 PM, Shad Kirmani 
>>> <sxk5292 AT cse.psu.edu>
>>>  wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am trying to build charm++ with SMP support. My machine specifications
>>>> are:
>>>>
>>>> [sxk5292@cyberstar84
>>>>  test]$ uname -a
>>>> Linux cyberstar84.hpc.rcc.psu.edu 2.6.18-274.7.1.el5 #1 SMP Mon Oct 17
>>>> 11:57:14 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> Please also visit http://www.ics.psu.edu/infrast/specs.html for more
>>>> detailed specifications.
>>>>
>>>> To build charm++, I chose the version mpi-linux-x86_64 with smp option
>>>> set.
>>>>
>>>> When I run my code I get the following output from charm++:
>>>> [sxk5292@cyberstar84
>>>>  test]$ ./pbsall.sh
>>>> Charm++> Running on MPI version: 2.1
>>>> Charm++> level of thread support used: MPI_THREAD_SINGLE (desired:
>>>> MPI_THREAD_FUNNELED)
>>>> Charm++> Running on SMP mode, 1 worker threads per process
>>>> Charm++> The comm. thread both sends and receives messages
>>>> Converse/Charm++ Commit ID: v6.3.0-1293-g7f245d0
>>>> Warning> Randomization of stack pointer is turned on in kernel.
>>>> ------------- Processor 14 Exiting: Caught Signal ------------
>>>> Signal: 11
>>>> ------------- Processor 49 Exiting: Caught Signal ------------
>>>> Signal: 11
>>>>
>>>> --------------------------------------------------------------------------
>>>> mpirun noticed that process rank 14 with PID 21394 on node cyberstar83
>>>> exited on signal 11 (Segmentation fault).
>>>>
>>>> --------------------------------------------------------------------------
>>>> ...
>>>> ...
>>>>
>>>> I have tried building charm++ with the following command line arguments:
>>>> ./build charm++ mpi-linux-x86_64 smp
>>>> and
>>>> ./build charm++ mpi-linux-x86_64 smp -DCMK_SMP=1 -DCMK_MPI_INIT_THREAD=1
>>>>
>>>> Can anybody please help with the command line arguments to build
>>>> charm++.
>>>>
>>>> Thanks,
>>>> Shad
>>>>
>>>>
>>>> _______________________________________________
>>>> charm mailing list
>>>> charm AT cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/charm
>>>>
>>>> _______________________________________________
>>>> ppl mailing list
>>>> ppl AT cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/ppl
>>>>
>>>
>>
>
>
> _______________________________________________
> charm mailing list
> charm AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/charm
>
> _______________________________________________
> ppl mailing list
> ppl AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/ppl
>