Charm++ parallel programming system

[Sam White <white67 AT illinois.edu>]

I'm not sure about the runtime errors, I'd need some more info on them.

The AMPI + Charm++ interop example should be enough to get it working from a Group and no migration. I am going to work on cleaning up that example some today.

-Sam

On Wed, Jul 18, 2018 at 10:51 AM, Jozsef Bakosi <jbakosi AT lanl.gov> wrote:

Hi Sam,

Thanks a lot for the detailed info. Please see more comments below inline.

On 07.17.2018 16:16, Sam White wrote:
> Interoperating AMPI with Charm++ in the way that you would like to is
> definitely something we are interested in, but we do not support it fully
> yet. Ideally, you'd be able to build your TPLs on AMPI, then make calls
> directly into those libraries from regular chare array elements, with a
> one-to-one mapping from chare array elements to AMPI ranks. In practice,
> this is a bit complicated because AMPI must create user-level threads for
> AMPI ranks to run on, so it can suspend and resume them to support the
> blocking semantics of MPI and also to migrate the thread stacks. AMPI also
> binds chare arrays to these user-level threads so that AMPI's internal
> state migrates along with the threads. AMPI's implementation currently is
> tied to it managing its own user-level threads in this way.

So far, we managed to build all of our MPI TPLs on top of AMPI. (I understand
that this does not necessarily mean they will work, especially, when migrated,
but it is a first step.)

We do get various runtime errors, somewhat surprisingly, even from those
executables that do not use MPI libraries. In particular, it appears that the
run is aborted without error randomly but always prematurely -- it as if someone
would have called finalize() on the whole run at random times. I'm not sure why
this could be, have you heard of a similar symptom porting apps from Charm++'s
MPI to a non-MPI communication layer?

The way you describe AMPI/Charm++ interoperation, using chare arrays bound to
AMPI ranks, would indeed be ideal and very exciting for us because we just
cannot do without MPI third-party libraries. However, it would also be already a
great intermediate step if we could make calls to MPI libraries from chare
groups and not migrate AMPI ranks.

> We have a hacky proof of concept of AMPI + Charm++ interoperation, wherein
> you can launch 'N' chare array elements and 'M' AMPI ranks (for arbitrary N
> and M) and send messages between the two collections, here:
> https://charm.cs.illinois.edu/gerrit/#/c/charm/+/4366/
>
> This means that the chare array elements cannot themselves make MPI calls,
> but they can send messages to MPI ranks which can then make those calls.

Nice! Do you think that example would work from groups and turning off AMPI
migration?

> There is also more work to be done to bind the MPI ranks to the chare array
> elements. We have plans for that in the future, and can stay in touch with
> you regarding updates on it. It's certainly motivating for us to hear from
> users who want this feature too.

Definitely, please keep me in the loop on this.

Thanks,
Jozsef

> On Mon, Jul 16, 2018 at 4:34 PM, Jozsef Bakosi <jbakosi AT lanl.gov> wrote:
>
> > Hi folks,
> >
> > I'm converting a Charm++ app that has so far successfully interoperated with
> > MPI using time division, previously built using Charm++ on top of MPI
> > (mpi-linux-x86_64), into now using Charm++ and AMPI (netlrts-linux-x86_64).
> >
> > As a new AMPI user, I have a couple of questions:
> >
> > 1. I noticed that when I try to include mpi-interoperate.h, I get:
> >
> > #error "Trying to compile Charm++/MPI interoperation against AMPI built atop
> > Charm++"
> >
> > I guess this means that AMPI and MPI interoperation is not intended to be
> > used in such a way. If that is so, how should I interoperate with code that
> > directly calls MPI functions on top of Charm++ code that is mostly written
> > using native Charm++ but also contains direct MPI calls?
> >
> > This is a test harness that has a main Charm++ module as well as a main()
> > routine which calls MPI_Init() followed by CharmLibInit() and
> > CharmLibExit(). In particular, can I no longer call CharmLibInit() and
> > CharmLibExit()? I did comment out these calls together with including
> > mpi-interoperate.h, and then the MPI-part of the code, the one in main(),
> > does not run at all. How should I do MPI interoperation within native
> > Charm++ that is built with AMPI?
> >
> > I bit more background on my motivation: I so far have been using Charm++ on
> > top of MPI because I want to use a number of MPI-only third-party libraries.
> > (I interface these TPLs from within chare groups and they work great.) Now I
> > would like to explore building Charm++ and AMPI and use the AMPI MPI
> > libraries to link MPI in so that (1) I don't need MPI, (2) use the advanced
> > AMPI load balancing features to do thread migration within the MPI-only
> > libraries. So far, I managed to build all my MPI TPLs with AMPI as well as
> > all my executables, but ran into the above issue. (I have not yet tried
> > running those executables that have calls to the MPI-only libraries.)
> >
> > Thanks for the insight in advance, Jozsef