Charm++ parallel programming system

[Nikhil Jain <nikhil.life AT gmail.com>]

Hi Jozsef,

Yes, those examples appear to be that way and there is a good reason -
our hands are tied due to initialization. MPI_Init is a bit adamant
about capturing resources it wants and hence it has to be called
before Charm is initialized. However, this is a minor limitation - for
interoperation, a predominantly Charm++ code should also start in a
similar manner and then switch to Charm++ for most parts.

Lets go through an example: tarball of an application using MPI-IO is
at http://charm.cs.uiuc.edu/~nikhil/episimdemics_interop.tgz. Don't
worry about the overall size, we will focus on specifics.

Look at interop/episim_io.cpp - this shows how the execution will
begin in MPI but switch to Charm++ for execution. This example passes
a null argument newComm for the communicator since I primarily meant
this to run on BG/Q using non-mpi layer (which is good for
performance). If you are using MPI layer of Charm++, newComm should be
duplicated from MPI_COMM_WORLD.

When CharmLibInit is called, the RTS create the main chare which is
part of the mainmodule declared in Global/main.ci. In non-interop set
up, the mainchare is created by the RTS without the need for any
explicit invocation (since Charm++ provides a main in that case). From
there on, the program progresses like any non-interop Charm++ code. As
I said earlier, this tweak is needed because MPI_Init needs to be
called before Charm++ init.

Note that, for such cases, its best you provide the mainmodule and
mainchares and pass "-mpi -nomain-module" flag during compilation
(Makefile-interop); if you don't provide them, the RTS will provide
its own mainmodule.

For other usage, interop/MPIIO_Helper.cpp defines the MPI functions
that are used by the code. They are invoked from various places in
ReadPersonData.h, ReadVisitData.h etc inside File directory. Note that
in this case, since MPI is secondary and I have coded them to avoid
any deadlock, I do not need to exit Charm++ scheduler etc. The calls
are simply made during one of Charm++'s entry method. Hope it helps.

If you want, I can set up a telecon with you to explain these and
answer other questions you may have.

Thanks
Nikhil

On Sun, May 4, 2014 at 10:20 PM, Jozsef Bakosi 
<jbakosi AT gmail.com>
 wrote:
> Hi Phil,
>
> Thanks for your answers, some questions/comments below inline.
>
> On Mon, Apr 28, 2014 at 4:14 PM, Phil Miller 
> <mille121 AT illinois.edu>
>  wrote:
>>
>> Hi Jozsef,
>>
>> There's some documentation on how to interoperate MPI and Charm++ code in
>> our manual:
>> http://charm.cs.illinois.edu/manuals/html/charm++/25.html
>
>
> I have seen this, but I thought the functionality that section describes is
> for using Charm++ libraries in MPI programs. What I'm after is the opposite:
> using a Charm++ program with MPI libraries, i.e., a predominantly Charm++
> application using, e.g., an MPI-based library for, e.g., I/O.
>>
>>
>> There was also a talk by Nikhil Jain at last year's Charm++ Workshop on
>> exactly this topic:
>> http://charm.cs.illinois.edu/workshops/charmWorkshop2013/program.php
>
>
> Those slides are interesting and do shed some light on the Charm++/MPI
> interoperation - using two main approaches: AMPI for existing MPI apps and
> (gradually) converting MPI apps to Charm++ apps. I'm in the (lucky)
> situation that I'm designing a code from scratch and I only want to use MPI
> if I have to: I would like to use Charm++ predominantly and use some
> existing well-established (MPI) libraries for e.g., the solution of linear
> systems and I/O.
>
>>>
>>> I wonder if there are examples (advantages, disadvantages, etc) of using
>>> third-party libraries that use MPI within a Charm++ application.
>>
>>
>> There's a paper currently in submission that covers this sort of mixing in
>> both directions (MPI libraries in predominantly Charm++ application, 
>> Charm++
>> libraries in predominantly MPI application)
>>
>> In the Charm++ repository, there are some code examples in
>> examples/charm++/mpi-coexist and examples/charm++/mpi-charmcollide
>
>
> I have looked at those two examples - thanks for drawing my attention to
> them. Don't they both demonstrate Charm++/MPI interoparability from the
> viewpoint of a predominantly MPI application using Charm++ functionality as
> a library? (I may not be familiar with Charm++ enough, so I will revisit
> these examples.)
>
>>
>> At least these various combinations of software have been tried
>> experimentally:
>> - Charm++ application with MPI IO library
>> - Charm++ application with MPI FFTW
>
>
> Cool! These seem like exactly what I'm looking for. BTW, I managed to build
> and run Hypre with AMPI (with Charm++ built on the mpi layer
> mpi-linux-x86_64), so it seems like this will work fine. I also timed one of
> the Hypre examples -- using algebraic multigrid as a preconditioner solving
> a simple cartesian-mesh-discretized Poisson equation -- with pure-MPI and
> Charm++/AMPI on top of MPI and the performance of the two were very close:
> the AMPI version was actually even faster by about 0.2% (which is probably
> only statistical variability). (Note that I did no virtualization: I simply
> run using the same number of CPU cores with MPI and Charm++. This was
> interesting for me to see if there is any kind of performance hit putting
> another layer (Charm++ and AMPI) between an MPI application and the MPI
> runtime.)
>
>>
>> The paper I referenced above (which I'll leave it to the authors to decide
>> whether/how to share) says that yes, it's possible and worthwhile.
>
>
> I would be interested in those papers. Please let me know when they are
> available for public consumption.
>
>>
>> Exactly how the various pieces are compiled and executed depends on the
>> platform an application is going to run on. In general, the design is such
>> that MPI libraries are natively usable, without any use of AMPI.
>
>
> That, I believe, answers my question. I will try to use MPI libraries then
> that way: simply compile them as they are usually compiled and link them to
> a Charm++ application.
>
> Thanks for your time,
> Jozsef
>
>



-- 
"Success may ditch you most of the times, but if you ditch it once :
your job is done."
Nikhil Jain, 
nikhil.life AT gmail.com,
 +1-217-979-0918