Skip to Content.
Sympa Menu

charm - Re: [charm] Fwd: Use of MPI third-party libraries within a Charm++ application

charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] Fwd: Use of MPI third-party libraries within a Charm++ application


Chronological Thread 
  • From: Jozsef Bakosi <jbakosi AT gmail.com>
  • To: Phil Miller <mille121 AT illinois.edu>
  • Cc: Nikhil Jain <nikhil.life AT gmail.com>, "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
  • Subject: Re: [charm] Fwd: Use of MPI third-party libraries within a Charm++ application
  • Date: Fri, 27 Feb 2015 15:20:01 -0700
  • List-archive: <http://lists.cs.uiuc.edu/pipermail/charm/>
  • List-id: CHARM parallel programming system <charm.cs.uiuc.edu>

Alright, I have figured out what happened.

I had an MPI call, (inside a Zoltan call) that expects to be called from all MPI ranks, inside one of the entry methods of my main chare. That was doing the Zoltan call only on a single PE, and I assume, thus it ended up in an infinite loop waiting for its counterpart(s) to synchronize.

As a solution, I am putting my Zoltan (MPI) call inside the main() function which, I assume, is guaranteed to execute on all MPI ranks after MPI was initialized properly. And since, for now, I only want time-sharing between Charm++ and the MPI library, I do not have to do anything fancy with the MPI communicator: both the MPI library and Charm++ use MPI_COMM_WORLD.

Thanks for the help,
Jozsef

On Tue, Feb 24, 2015 at 10:03 PM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:


On Tue, Feb 24, 2015 at 9:55 PM, Phil Miller <mille121 AT illinois.edu> wrote:
Could you please post your complete Charm++ build command, and job script? I have some suspicions of what's going on, but they'll be easier to consider with that information.


Sure, here they are

$ ./build charm++ mpi-linux-x86_64 mpicxx --build-shared -j8 -stdlib=libc++ -lc++abi -g
$ ./charmrun +p4 <executable>

Also, I'm using a manually-built OpenMPI 1.8 with Clang 3.5 as mpi-wrappers and underlying compiler, respectively. Please let me know if you need more information.
 
On Tue, Feb 24, 2015 at 10:52 PM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:
Yes, you were right, it was most likely picking up AMPI's mpi.h, because I was also getting linker errors related to missing AMPI init calls. For now, I simply removed AMPI from my Charm++ build, and now it seems like it picks up my OpenMPI's mpi.h, thus the compilation and linking go fine. That's good progress, thanks, Phil.

Now, when I insert some Zoltan calls that access the MPI communicator, I get hangs if I run on more than one PE, e.g., charmrun +p2. Here is my main(), which does NOT hang. Note that I ended up putting Zoltan_Create() BEFORE CharmLibInit(). If I move that after CharmLibInit(), I get the hang with +p2. This call sequence would be okay, but I'm afraid I'll get into trouble with other Zoltan calls that access the MPI communicator which must be after CharmLibInit(), e.g., the call to partition. If I want to do a simple time-division between MPI and Charm++ execution, do I have to enclose the MPI-library calls in between StopCharmScheduler() and StartCharmScheduler(), as discussed in Jain et al, "On Interoperation of User-driven and System-driven Parallel Languages", 2014?

int main( int argc, char **argv ) {
  int peid, numpes;

  MPI_Init( &argc, &argv );
  MPI_Comm_rank( MPI_COMM_WORLD, &peid );
  MPI_Comm_size( MPI_COMM_WORLD, &numpes );

  float ver = 0.0;
  Zoltan_Initialize( 0, nullptr, &ver );

  struct Zoltan_Struct *z;
  z = Zoltan_Create( MPI_COMM_WORLD );

  CharmLibInit( MPI_COMM_WORLD, argc, argv );
  MPI_Barrier( MPI_COMM_WORLD );

  CharmLibExit();

  Zoltan_Destroy( &z );

  MPI_Finalize();

  return 0;  
}

On Tue, Feb 24, 2015 at 4:24 PM, Phil Miller <mille121 AT illinois.edu> wrote:
After a bit more examination, it's also possible that you've compiled with the host mpicc, but it's picking up AMPI's mpi.h and ending up with a very confused build. If that's the case, Nikhil will probably have to comment on the right way to address it. His response may be delayed, as he presents his thesis proposal later this week.

On Tue, Feb 24, 2015 at 5:22 PM, Phil Miller <mille121 AT illinois.edu> wrote:
It looks like you're compiling your code as AMPI, our implementation of MPI built on top of Charm++. That is an alternative to the native MPI interoperation facilities. For MPI-Charm++ interop, the MPI code should be compiled with the host's mpicc or equivalent.

On Tue, Feb 24, 2015 at 5:18 PM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:
Yeah, I noticed that too, and I tried passing MPI_COMM_WORLD directly to CharmLibInit as well as duplicating the communicator and passing that one too. The errors, respectively, are:

---------------------
[karman:21834] *** An error occurred in MPI_Comm_dup
[karman:21834] *** reported by process [2918055937,0]
[karman:21834] *** on communicator MPI_COMM_WORLD
[karman:21834] *** MPI_ERR_COMM: invalid communicator
[karman:21834] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[karman:21834] ***    and potentially your MPI job)

--------------------
------------- Processor 0 Exiting: Caught Signal ------------
Reason: Segmentation fault
[0] Stack Traceback:
  [0:0]   [0x86bc87]
  [0:1] +0x35180  [0x7131180]
  [0:2] MPI_Comm_dup+0x4e  [0x674f41e]
  [0:3] CharmLibInit+0x26  [0x862e26]
  [0:4] _Z13AMPI_Main_cppiPPc+0x67  [0x563f37]
  [0:5] AMPI_Fallback_Main+0x1b  [0x5c42cb]
  [0:6] _ZN17MPI_threadstart_t5startEv+0x64  [0x5effd4]
  [0:7] AMPI_threadstart+0x2e  [0x5c451e]
  [0:8]   [0x5b7bee]
  [0:9] CthStartThread+0x5a  [0x7293fa]
  [0:10] +0x45f60  [0x7141f60]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0 
with errorcode 1.
---------------------

And: now I took out the Zoltan calls and so now this is coming from CharmLibInit(). Any ideas?

On Tue, Feb 24, 2015 at 4:10 PM, Phil Miller <mille121 AT illinois.edu> wrote:
It looks like 'newComm' is never initialized in that code. Should there be a call to MPI_Comm_dup there? You could actually just pass MPI_COMM_WORLD to CharmLibInit, since it will duplicate its argument internally.

On Tue, Feb 24, 2015 at 5:05 PM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:
Thanks Nikhil,

I finally got around to try this. I followed your instructions in interop/episim_io.cpp and successfully built my executable with the main() function initializing MPI and then calling CharmLibInit(). This is all good and working.

The problem I'm having now is the following. I'd like to do an initial static mesh partitioning using the Zoltan library. When I try to make a call to Zoltan_Create() passing an MPI communicator, I get a segfault. Here is what I do (I copied almost exactly your main function to my code and added calls to initialize and finalize Zoltan):

int main(int argc, char **argv) {
  int peid, numpes;
  MPI_Comm newComm;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &peid);
  MPI_Comm_size(MPI_COMM_WORLD, &numpes);

  float ver = 0.0;
  Zoltan_Initialize( 0, nullptr, &ver );

  struct Zoltan_Struct *zz;
  zz = Zoltan_Create( MPI_COMM_WORLD );

  Zoltan_Destroy( &zz );

  CharmLibInit(newComm, argc, argv);
  
  MPI_Barrier(MPI_COMM_WORLD);

  CharmLibExit();

  MPI_Finalize();

  return 0;  
}


Here is the error message. It seems like the problem comes from Zoltan_Create when it is internally trying to create a duplicate of the MPI communicator:

==5587== Invalid read of size 4
==5587==    at 0x67281DB: ompi_comm_dup_with_info (in /opt/openmpi/1.8/clang/system/lib/libmpi.so.1.5.0)
==5587==  Address 0x10ec8348e5894865 is not stack'd, malloc'd or (recently) free'd
==5587== 
------------- Processor 0 Exiting: Caught Signal ------------
Reason: Segmentation fault
[0] Stack Traceback:
  [0:0]   [0x86bca7]
  [0:1] +0x35180  [0x7131180]
  [0:2] ompi_comm_dup_with_info+0x2b  [0x67281db]
  [0:3] MPI_Comm_dup+0x92  [0x674f462]
  [0:4] Zoltan_Create+0x109  [0x638369]
  [0:5] _Z13AMPI_Main_cppiPPc+0x2db  [0x563e2b]
  [0:6] AMPI_Fallback_Main+0x1b  [0x5c42eb]
  [0:7] _ZN17MPI_threadstart_t5startEv+0x64  [0x5efff4]
  [0:8] AMPI_threadstart+0x2e  [0x5c453e]
  [0:9]   [0x5b7c0e]
  [0:10] CthStartThread+0x5a  [0x72941a]
  [0:11] +0x45f60  [0x7141f60]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0 
with errorcode 1.

Should Zoltan_Create() be called after CharmLibInit()? I would like that better, but I had no luck with that either. I also tried duplicating the MPI communicator (as you suggested earlier) and passed that communicator to Zoltan_Create, but that also segfaults. I'm not sure what I'm missing.

Another, somewhat related, question: Since I want to use Zoltan to give me an initial static partitioning of a mesh, after which I will not need Zoltan at all, can I somehow call Zoltan inside from Charm++ (which also has a load balancer option based on Zoltan), or that functionality is not exposed to the application that way? That way would be easier for me since I would not have to deal with explicitly interfacing Zoltan in my application.

Thanks,
Jozsef

On Tue, May 6, 2014 at 10:15 AM, Nikhil Jain <nikhil.life AT gmail.com> wrote:
Hi Jozsef,

Yes, those examples appear to be that way and there is a good reason -
our hands are tied due to initialization. MPI_Init is a bit adamant
about capturing resources it wants and hence it has to be called
before Charm is initialized. However, this is a minor limitation - for
interoperation, a predominantly Charm++ code should also start in a
similar manner and then switch to Charm++ for most parts.

Lets go through an example: tarball of an application using MPI-IO is
at http://charm.cs.uiuc.edu/~nikhil/episimdemics_interop.tgz. Don't
worry about the overall size, we will focus on specifics.

Look at interop/episim_io.cpp - this shows how the execution will
begin in MPI but switch to Charm++ for execution. This example passes
a null argument newComm for the communicator since I primarily meant
this to run on BG/Q using non-mpi layer (which is good for
performance). If you are using MPI layer of Charm++, newComm should be
duplicated from MPI_COMM_WORLD.

When CharmLibInit is called, the RTS create the main chare which is
part of the mainmodule declared in Global/main.ci. In non-interop set
up, the mainchare is created by the RTS without the need for any
explicit invocation (since Charm++ provides a main in that case). From
there on, the program progresses like any non-interop Charm++ code. As
I said earlier, this tweak is needed because MPI_Init needs to be
called before Charm++ init.

Note that, for such cases, its best you provide the mainmodule and
mainchares and pass "-mpi -nomain-module" flag during compilation
(Makefile-interop); if you don't provide them, the RTS will provide
its own mainmodule.

For other usage, interop/MPIIO_Helper.cpp defines the MPI functions
that are used by the code. They are invoked from various places in
ReadPersonData.h, ReadVisitData.h etc inside File directory. Note that
in this case, since MPI is secondary and I have coded them to avoid
any deadlock, I do not need to exit Charm++ scheduler etc. The calls
are simply made during one of Charm++'s entry method. Hope it helps.

If you want, I can set up a telecon with you to explain these and
answer other questions you may have.

Thanks
Nikhil

On Sun, May 4, 2014 at 10:20 PM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:
> Hi Phil,
>
> Thanks for your answers, some questions/comments below inline.
>
> On Mon, Apr 28, 2014 at 4:14 PM, Phil Miller <mille121 AT illinois.edu> wrote:
>>
>> Hi Jozsef,
>>
>> There's some documentation on how to interoperate MPI and Charm++ code in
>> our manual:
>> http://charm.cs.illinois.edu/manuals/html/charm++/25.html
>
>
> I have seen this, but I thought the functionality that section describes is
> for using Charm++ libraries in MPI programs. What I'm after is the opposite:
> using a Charm++ program with MPI libraries, i.e., a predominantly Charm++
> application using, e.g., an MPI-based library for, e.g., I/O.
>>
>>
>> There was also a talk by Nikhil Jain at last year's Charm++ Workshop on
>> exactly this topic:
>> http://charm.cs.illinois.edu/workshops/charmWorkshop2013/program.php
>
>
> Those slides are interesting and do shed some light on the Charm++/MPI
> interoperation - using two main approaches: AMPI for existing MPI apps and
> (gradually) converting MPI apps to Charm++ apps. I'm in the (lucky)
> situation that I'm designing a code from scratch and I only want to use MPI
> if I have to: I would like to use Charm++ predominantly and use some
> existing well-established (MPI) libraries for e.g., the solution of linear
> systems and I/O.
>
>>>
>>> I wonder if there are examples (advantages, disadvantages, etc) of using
>>> third-party libraries that use MPI within a Charm++ application.
>>
>>
>> There's a paper currently in submission that covers this sort of mixing in
>> both directions (MPI libraries in predominantly Charm++ application, Charm++
>> libraries in predominantly MPI application)
>>
>> In the Charm++ repository, there are some code examples in
>> examples/charm++/mpi-coexist and examples/charm++/mpi-charmcollide
>
>
> I have looked at those two examples - thanks for drawing my attention to
> them. Don't they both demonstrate Charm++/MPI interoparability from the
> viewpoint of a predominantly MPI application using Charm++ functionality as
> a library? (I may not be familiar with Charm++ enough, so I will revisit
> these examples.)
>
>>
>> At least these various combinations of software have been tried
>> experimentally:
>> - Charm++ application with MPI IO library
>> - Charm++ application with MPI FFTW
>
>
> Cool! These seem like exactly what I'm looking for. BTW, I managed to build
> and run Hypre with AMPI (with Charm++ built on the mpi layer
> mpi-linux-x86_64), so it seems like this will work fine. I also timed one of
> the Hypre examples -- using algebraic multigrid as a preconditioner solving
> a simple cartesian-mesh-discretized Poisson equation -- with pure-MPI and
> Charm++/AMPI on top of MPI and the performance of the two were very close:
> the AMPI version was actually even faster by about 0.2% (which is probably
> only statistical variability). (Note that I did no virtualization: I simply
> run using the same number of CPU cores with MPI and Charm++. This was
> interesting for me to see if there is any kind of performance hit putting
> another layer (Charm++ and AMPI) between an MPI application and the MPI
> runtime.)
>
>>
>> The paper I referenced above (which I'll leave it to the authors to decide
>> whether/how to share) says that yes, it's possible and worthwhile.
>
>
> I would be interested in those papers. Please let me know when they are
> available for public consumption.
>
>>
>> Exactly how the various pieces are compiled and executed depends on the
>> platform an application is going to run on. In general, the design is such
>> that MPI libraries are natively usable, without any use of AMPI.
>
>
> That, I believe, answers my question. I will try to use MPI libraries then
> that way: simply compile them as they are usually compiled and link them to
> a Charm++ application.
>
> Thanks for your time,
> Jozsef
>
>



--
"Success may ditch you most of the times, but if you ditch it once :
your job is done."
Nikhil Jain, nikhil.life AT gmail.com, +1-217-979-0918


_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm












Archive powered by MHonArc 2.6.16.

Top of Page