Skip to Content.
Sympa Menu

charm - Re: [charm] [ppl] Using TAU with AMPI

charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] [ppl] Using TAU with AMPI


Chronological Thread 
  • From: Xoan Carlos Pardo Martinez <xoanpardo AT gmail.com>
  • To: Chee Wai Lee <cheewai1972 AT gmail.com>
  • Cc: charm AT cs.uiuc.edu
  • Subject: Re: [charm] [ppl] Using TAU with AMPI
  • Date: Thu, 21 Jun 2012 18:22:14 +0200
  • List-archive: <http://lists.cs.uiuc.edu/pipermail/charm>
  • List-id: CHARM parallel programming system <charm.cs.uiuc.edu>

Hi Chee,

Sorry for this long time without notices from me.

During this time I´ve been doing many tests with Charm++, AMPI and TAU trying to understand the best way to make the measurements I'm interested in.
At first I was a bit disoriented by your comment about TAU MPI wrapper calls: "As a default, TAU wraps MPI calls. In the case of attempting to measure 
AMPI performance, we do not want to do that (it is actually unclear which MPI layer gets wrapped in this case) but instead interpret the calls according 
to what the Charm++ runtime tells TAU." I suspected that If I wanted comparable AMPI measures with other MPI implementations form the application 
point-of-view, I had to use the TAU MPI wrapping interface over Charm++ compiled without any underlying MPI support. So I've decided to try that approach.

The Tau MPI wrapper library expects that the underlying MPI implementation provides an implementation of the PMPI interface (MPI Profiling interface).
Most MPI functions in the wrapper library have implementations like this:

int  MPI_fun( ... )
{
  int  returnVal;
  int typesize;
  
  TAU_PROFILE_TIMER(tautimer, "MPI_fun()",  " ", TAU_MESSAGE);
  TAU_PROFILE_START(tautimer);

// ... some TAU stuff here ...

  returnVal = PMPI_fun( ... );

// ... some TAU stuff here ...

  TAU_PROFILE_STOP(tautimer);
  return returnVal;
}

That is, they are calls to PMPI functions surrounded by TAU code to start/stop trace of events. Unfortunately AMPI doesn't provide an PMPI interface. So
I had to modify ampi.h in the src/libs/ck-libs/ampi Charm++ directory. For every MPI function I defined the following:

#ifdef TAU_MPI
   #define PMPI_fun AMPI_fun
   int MPI_fun( ... );
#else
   #define MPI_fun AMPI_fun
#endif
int AMPI_fun( ... );

This way, when compiling with TAU, PMPI functions are renamed as AMPI functions, and TAU will be tracing AMPI function calls.

To test the modifications I follow these steps (really I did LOTS of tests until it works):

1. Build Charm++ without underlying MPI support
2. Build AMPI (without defining TAU_MPI)
3. Build TAU using AMPI and -TRACE and -MPITRACE options and without using any thread library.
4. Build Charm++ Tau tracing support using the TAU libs and makefile created in step 3
5. Build an example MPI application (PI calculation) with TAU instrumentation and AMPI. I linked it with the -tracemode Tau option.

The good news are that it works. I can execute the application and get the TAU trace files with performance information about the MPI function calls.
Bad news are that from time to time execution crashes for different reasons. Sometimes I get error messages from Charm++ and others I get signal
exceptions apparently caused by TAU. I´ve noticed also that executing with the ++ppn option (with a value different from 1) doesn´t work at all.

For me it´s really hard to know what can be causing this behaviour. I wonder if anyone from the Charm (or TAU) team could give me any assistance
If I provide debug information of failed executions.

Best regards
----
Xoan C. Pardo
Computer Architecture Group
University of A Coruña - Spain



Em 13/04/2012, às 19:18, Chee Wai Lee escreveu:

Hi Xoan,

I do not believe anyone has attempted to use TAU profiling (via the Charm++-TAU measurement interface) on AMPI so far. I should be able to help you navigate through some of the issues and perhaps adapt the framework to enable what you desire if some functionality is missing.

The way TAU interfaces with the Charm++ runtime to measure the performance of Charm++ events (on top of which AMPI is built) is through callback hooks provided by the runtime system. You will need to follow steps similar to the following:

http://www.nic.uoregon.edu/tau-wiki/Guide:NAMDTAU

The only requirements on the configuration and building of TAU is that it should match the Charm++ build as closely as possible (the architecture, the compilers and the use of any underlying MPI layers). In your case, the use of "-no-trace-mpi" is critical. As a default, TAU wraps MPI calls. In the case of attempting to measure AMPI performance, we do not want to do that (it is actually unclear which MPI layer gets wrapped in this case) but instead interpret the calls according to what the Charm++ runtime tells TAU.

You may contact me via skype if you need any interactive assistance on the matter. What you are doing is new as far as I know and might require some work to get going correctly. Thanks!

Best Regards,
Chee Wai Lee
University of Oregon

On Apr 10, 2012, at 6:23 AM, Xoan Carlos Pardo Martinez wrote:

Hi,

I´m interested in making some performance measurements using different MPI implementations. I´m using TAU as profile tool and AMPI is one of the implementations i´m interested in.

My question is about the right way to configure/compile TAU to use it with AMPI. Before building the Tau target in CHARM, is it necessary to compile and configure TAU to use the commands/headers/libraries in the AMPI directory? If so, is it also necessary to specify the -charm building option to use CHARM threads in TAU?

Thanks for your assistance

Best regards
----
Xoan C. Pardo
Computer Architecture Group
University of A Coruña - Spain
_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm

_______________________________________________
ppl mailing list
ppl AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/ppl






Archive powered by MHonArc 2.6.16.

Top of Page