Skip to Content.
Sympa Menu

charm - Re: [charm] Use of MPI third-party libraries within a Charm++ application

charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] Use of MPI third-party libraries within a Charm++ application


Chronological Thread 
  • From: Phil Miller <mille121 AT illinois.edu>
  • To: Jozsef Bakosi <jbakosi AT gmail.com>
  • Cc: "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
  • Subject: Re: [charm] Use of MPI third-party libraries within a Charm++ application
  • Date: Mon, 28 Apr 2014 17:14:41 -0500
  • List-archive: <http://lists.cs.uiuc.edu/pipermail/charm/>
  • List-id: CHARM parallel programming system <charm.cs.uiuc.edu>

Hi Jozsef,

There's some documentation on how to interoperate MPI and Charm++ code in our manual:
http://charm.cs.illinois.edu/manuals/html/charm++/25.html


There was also a talk by Nikhil Jain at last year's Charm++ Workshop on exactly this topic:
http://charm.cs.illinois.edu/workshops/charmWorkshop2013/program.php

I wonder if there are examples (advantages, disadvantages, etc) of using third-party libraries that use MPI within a Charm++ application.

There's a paper currently in submission that covers this sort of mixing in both directions (MPI libraries in predominantly Charm++ application, Charm++ libraries in predominantly MPI application)

In the Charm++ repository, there are some code examples in examples/charm++/mpi-coexist and examples/charm++/mpi-charmcollide
 
For example, are you aware of anyone developing a Charm++ application that uses an existing linear solver + preconditioner library built on top of MPI, such as PETSc; Trilinos' Belos, IfPack, ML, etc; or Hypre?

Is this possible? Is it worth it? If so, what is the best way to build the library and link to the Charm++ application? Must the third-party MPI library be compiled using AMPI? Or will it work simply compiling with e.g., OpenMPI (as usual), and link to a Charm++ application built with charmc?

Also, should Charm++ be built with an MPI-based network interface in that case so that the MPI interface and library is available for the third-party library or is it possible to use a non-MPI network layer for Charm++ and still use the MPI library?


Another, somewhat related, question is the use of a parallel I/O library, such as ADIOS, within a Charm++ application. ADIOS is also built on MPI by default. What would be the best way to go about using such a library within a Charm++ application?

Some of my labmates can answer this in more detail (and perhaps have in private messages), but I'll give the parts of the answer(s) I know personally.

At least these various combinations of software have been tried experimentally:
- Charm++ application with MPI IO library
- Charm++ application with MPI FFTW
- MPI solver library using Charm++ code for an internal step
- MPI application using Charm++ for distributed sorting
 
The paper I referenced above (which I'll leave it to the authors to decide whether/how to share) says that yes, it's possible and worthwhile.

Exactly how the various pieces are compiled and executed depends on the platform an application is going to run on. In general, the design is such that MPI libraries are natively usable, without any use of AMPI. On the current high-end proprietary networks (IBM's Blue Gene Q and Cray's Gemini and Ares), Charm++ and MPI can concurrently access the native network stack directly. I'm less certain of the state of Charm++'s internals on commodity Infiniband and Ethernet based clusters.

On a broader note, the annual Charm++ Workshop is going on this week. This means that mail responses may be slower than we'd usually like to maintain. I don't think any of the scheduled talks this year will touch on our interoperability efforts, but I'm not certain of that. At any rate, there will be a live webcast and eventually posted video recordings.

http://charm.cs.illinois.edu/charmWorkshop/index.php

I hope the above helps, and will ask my colleagues to follow up.


Phil



Archive powered by MHonArc 2.6.16.

Top of Page