Skip to Content.
Sympa Menu

charm - [charm] MPI_THREAD_SINGLE in OpenAtom

charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

[charm] MPI_THREAD_SINGLE in OpenAtom


Chronological Thread 
  • From: Jeff Hammond <jeff.science AT gmail.com>
  • To: "Kale, Laxmikant V" <kale AT illinois.edu>
  • Cc: "charm AT lists.cs.illinois.edu" <charm AT lists.cs.illinois.edu>, "openatom-ppl AT lists.illinois.edu" <openatom-ppl AT lists.illinois.edu>
  • Subject: [charm] MPI_THREAD_SINGLE in OpenAtom
  • Date: Thu, 24 Mar 2016 11:45:09 -0700

It's certainly not the issue here but MPI_THREAD_SINGLE with Charm++ is likely technically incorrect use of MPI. This MPI thread level is a assertion that no threads are used at all anywhere, and implies that the MPI library can use thread-unsafe malloc etc. SINGLE is roughly equivalent to using xlc instead of xlc_r on IBM systems...

Most implementations won't see any issues here thanks to the ubiquity of Pthreads, but to be strictly correct, you should use FUNNELED at the minimum, and potentially SERIALIZED or MULTIPLE if threads are calling MPI with or without mutual exclusion, respectively. 

Note that threads in BLAS are enough to require the use of FUNNELED if one desires standard-compliant use of MPI. 

I apologize if OpenAtom, its dependencies and Charm++ are not using threads anywhere, in which case SINGLE is fine.

Sorry for the distraction, but MPI threading semantics are a pet of mine :-)

Jeff 

On Thursday, March 24, 2016, Kale, Laxmikant V <kale AT illinois.edu> wrote:
I am forwarding this to the charm++ mailing list for a speedy response. — -Sanjay

From: "Lukasz Orlowski (ACRC)" <lukasz-o AT acrc.a-star.edu.sg>
Date: Thursday, March 24, 2016 at 4:59 AM
To: Laxmikant Kale <kale AT illinois.edu>
Cc: "Marek T Michalewicz (ACRC)" <Michalewiczm AT acrc.a-star.edu.sg>
Subject: Requesting help running OpenAtom on Power8 platform

Hi Kale,

I’m working with Dr Marek Michalewicz, I’m having problems running OpenAtom on Power8 platform and he recommended I contact you, describing what problems I’m running into.

I compile charm++ with the following command:

$ ./build charm++ mpi-linux-ppc --with-production -j80

charm++ compiles properly. Then I follow the instruction to test a sample program:

$ cd mpi-linux-ppc/tests/charm++/simplearrayhello
$ make
$ ./charmrun ./hello

I get a message from setarch, that the ppc64le architecture is unrecognised. I am however able to run the sample program with:

$ mpirun -np 1  ./hello

Charm-FFT and OpenAtom compile properly again, but when I run a test as described on OpenAtom’s web page: http://charm.cs.illinois.edu/OpenAtom/download.shtml, I get the following warnings and seg fault at the end:

$ mpirun -np 4 ../../build-O3/OpenAtom tests/simple_minimization/cpaimd_config.p1 tests/simple_minimization/water.input 
Charm++> Running on MPI version: 3.0
Charm++> level of thread support used: MPI_THREAD_SINGLE (desired: MPI_THREAD_SINGLE)
Charm++> Running in non-SMP mode: numPes 4
Converse/Charm++ Commit ID: v6.7.0-1-gca55e1d
Warning> Randomization of stack pointer is turned on in kernel, thread migration may not work! Run 'echo 0 > /proc/sys/kernel/randomize_va_space' as root to disable it, or try run with '+isomalloc_sync'.  
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (160-way SMP).
Charm++> cpu topology info is gathered in 5.012 seconds.
Executing OpenAtom: BINARY - ../../build-O3/OpenAtom
Binary produced from source-tree at commit: 07b6e34e4f67d922fa4cfb2a5c288b4dddd6a5ac

==============================================================================
Starting Cpaimd-Charm-Driver Setup Phase
------------------------------------------------------------------------------
  Cpaimd-Charm-Driver running on 4 processors. 
At the beginning of the run user mem 345.875000 MB
  Reading Physics input from tests/simple_minimization/water.input

==============================================================================
Invoking PINY Interface
------------------------------------------------------------------------------


==============================================================================
Reading simulation input file tests/simple_minimization/water.input
------------------------------------------------------------------------------

Simulation param. allocation: 0.0024 Mbytes; Total memory: 0.0024 Mbytes
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$$$$$$
The recommended range of cp_nonloc_pme_order is 4-8 
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You are in danger! Your cp_min_update is off! 
Are you debugging some CS code? 
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$

$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
The true cp_mass_tau is 1581.14. The value set 25
is scaled by sqrt(cmass_cut_def/1000)=44.7214.
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$

$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You have requested more than 1000 spline points   
Are you certain this is what you would like to do?
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You have requested a cp run tolerence greater than  
10000 times the minimization tolerence 5 0.0001 
Are you certain this is what you would like to do?
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You have requested a write inst freq > 2000 steps 
Are you certain this is what you would like to do?
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You have requested a write position freq > 2000 steps 
Are you certain this is what you would like to do?
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
You have requested a write velocity freq > 2000 steps 
Are you certain this is what you would like to do?
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
For CP calculations, interatomic PE should be 
Calculated every step.  Setting the frequency to 1
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
Restart types other than restart_all employed
in parallel may effect comparisons with simulations 
performed in scalar or with different numbers of procs
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
CP-Restart types other than restart_all employed
in parallel may effect comparisons with simulations 
performed in scalar or with different numbers of procs
$$$$$$$$$$$$$$$$$$$$_warning_$$$$$$$$$$$$$$$$$$$$
@@@@@@@@@@@@@@@@@@@@_error_@@@@@@@@@@@@@@@@@@@@
ERROR: water already exists! (exiting)
@@@@@@@@@@@@@@@@@@@@_error_@@@@@@@@@@@@@@@@@@@@
[Partition 0][Node 0] End of program
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 33283 on node firestone exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

As you see I don’t use charmrun due to the issue with ppc64le, but rather mpirun by hand.

Would you be able to tell me what might be the issue or to whom should I talk about it?

Regards,
Lukasz Orlowski

Computational Scientist
A*STAR Computational Resource Centre
Phone: +65 6419 1306



--
Jeff Hammond
jeff.science AT gmail.com
http://jeffhammond.github.io/


  • [charm] MPI_THREAD_SINGLE in OpenAtom, Jeff Hammond, 03/24/2016

Archive powered by MHonArc 2.6.16.

Top of Page