Charm++ parallel programming system

[Phil Miller <mille121 AT illinois.edu>]

On Thu, Sep 8, 2011 at 13:19, Eric Bohm 
<ebohm AT illinois.edu>
 wrote:
> The Charm++ ibverbs layer is slightly specific to mellanox API, so I am
> not surprised that there are some problems using it on QLogic.  Given a
> pointer to a PSM API we could look it over and estimate the effort level
> required for a PSM version.

Sorry for not following up on this earlier, but I have something
useful to report. The next release of PSM (possibly just software
stack, not new hardware, though I'm unsure) is said to support the
ibverbs API directly, without emulation. The performance of our
ibverbs layer should get much better when that happens.

> On 09/08/2011 12:55 PM, Bennion, Brian wrote:
>> This is from one of our system admins and MPI gurus.  Does anyone have 
>> experience with the PSM libraries he is talking about?
>>
>> Brian
>> Sent: Thursday, September 08, 2011 10:42 AM
>> To: Bennion, Brian
>> Subject: Re: using ibverbs instead of mpi on sierra
>>
>> Hi Brian,
>> Yes, you're allowed to use ibverbs directly on sierra.  However, there
>> is a catch regarding ibverbs on sierra, muir, and ansel.  These machines
>> use the QLogic interconnect, which natively supports a library called
>> PSM or psm_infinipath.  The ibverbs library is available there, but it
>> gets relatively bad performance because it's emulated in software and
>> not supported directly in hardware like PSM is.  On our other systems,
>> like atlas and hera, ibverbs is supported directly in hardware so it
>> performs much better.  My advice on sierra is to either stick with MPI
>> (which runs over PSM there), or code to PSM directly.  Coding directly
>> to ibverbs on sierra could very well lead to the kind of the problem
>> you're seeing now.  Unfortunately, I don't have a good reference on
>> coding to PSM -- we just rely on the MPI vendors to figure it out.  If
>> you'd like, I could try to hunt down some PSM documentation.
>> -Adam
>>
>> Bennion, Brian wrote:
>>
>>>
>>> Hello Adam.
>>>
>>> I am curious as to whether users are allowed to compile and use the ib 
>>> layer instead of the mpi communications layer on sierra.
>>>
>>> I have compiled a version of charm++/namd2.8 that uses the ibverbs 
>>> communications layer and things start OK on sierra, but then random 
>>> socket closures kill my job at the very begining.
>>>
>>> There is a promise of 30% speedup using ib instead of mpi on infinban 
>>> clusters, hence my desire to use it.
>>>
>>> any help would be appreciated.
>>>
>>> Brian Bennion
>>>
>>>
>>
>>
>> _______________________________________________
>> charm mailing list
>> charm AT cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/charm
>
> _______________________________________________
> charm mailing list
> charm AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/charm
> _______________________________________________
> ppl mailing list
> ppl AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/ppl
>