Charm++ parallel programming system

[Jim Phillips <jim AT ks.uiuc.edu>]

OK, that might explain it.  Just stick with MPI on QLogic.  -Jim

On Thu, 8 Sep 2011, Bennion, Brian wrote:

> This is from one of our system admins and MPI gurus.  Does anyone have 
> experience with the PSM libraries he is talking about?
>
> Brian
> Sent: Thursday, September 08, 2011 10:42 AM
> To: Bennion, Brian
> Subject: Re: using ibverbs instead of mpi on sierra
>
> Hi Brian,
> Yes, you're allowed to use ibverbs directly on sierra.  However, there
> is a catch regarding ibverbs on sierra, muir, and ansel.  These machines
> use the QLogic interconnect, which natively supports a library called
> PSM or psm_infinipath.  The ibverbs library is available there, but it
> gets relatively bad performance because it's emulated in software and
> not supported directly in hardware like PSM is.  On our other systems,
> like atlas and hera, ibverbs is supported directly in hardware so it
> performs much better.  My advice on sierra is to either stick with MPI
> (which runs over PSM there), or code to PSM directly.  Coding directly
> to ibverbs on sierra could very well lead to the kind of the problem
> you're seeing now.  Unfortunately, I don't have a good reference on
> coding to PSM -- we just rely on the MPI vendors to figure it out.  If
> you'd like, I could try to hunt down some PSM documentation.
> -Adam
>
> Bennion, Brian wrote:
>
> > Hello Adam.
> >
> > I am curious as to whether users are allowed to compile and use the ib layer 
> > instead of the mpi communications layer on sierra.
> >
> > I have compiled a version of charm++/namd2.8 that uses the ibverbs 
> > communications layer and things start OK on sierra, but then random socket 
> > closures kill my job at the very begining.
> >
> > There is a promise of 30% speedup using ib instead of mpi on infinban 
> > clusters, hence my desire to use it.
> >
> > any help would be appreciated.
> >
> > Brian Bennion
>
>
> _______________________________________________
> charm mailing list
> charm AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/charm
> _______________________________________________
> ppl mailing list
> ppl AT cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/ppl