Charm++ parallel programming system

[Eric Bohm <ebohm AT illinois.edu>]

None of Charm++'s performance features are negatively impacted by using a different launcher.  Not using charmrun can impact debugging, visualization, or resilience management on some platforms.  But performance should not be affected. 

Using the mpi layer can lead to some performance loss.  Our large scale production runs generally use a lower level API build target as appropriate to the platform.

On 11/18/2014 11:04 AM, Scott Field wrote:

Hi,

  Thanks again for the suggestions and your informative, quick responses!

  Phil: I haven't yet explored the smp build option. The alternative method, which I tried first, worked. Later this week I will give smp builds a try. Thanks for dissuading me from using the nodelist approach.

  I have been able to run 16k-core jobs on Stampede using

./build charm++ mpi-linux-x86_64   mpicxx  -j16  --with-production

ibrun ./pgm

I wasn't even aware one could run a parallelized charm++ program without charmrun. Instead, I thought I was going to get 16k copies of standalone mode. This is the behaviour when using ibrun with a net-linux-.... charm++ build. Since I don't need a net-linux-... build (it was just to get me going with charm++) I'm happy to proceed without it.

Are any of the attractive charm++ features (asynchronous message passing, load balancing, fault tolerance, or any other runtime system feature) either missing or known to suffer a performance decrease when using ibrun or mpirun (with the mpi-... charm++ build) instead of charmrun? Or will the two be interchangeable in regard to performance and functionality?

Scott

On Thu, Nov 13, 2014 at 1:53 PM, Esteban Meneses-Rojas <emeneses AT pitt.edu> wrote:
   Scott,
   I am currently running some 16K-core (1024-node) jobs on Stampede
with Charm++.
   Here is the build I am using:
./build charm++ mpi-linux-x86_64
   And, this is how I run it:
ibrun ./pgm

On Wed, Nov 12, 2014 at 7:58 PM, Scott Field <sfield AT astro.cornell.edu> wrote:
> Hi Jim, Bilge and the rest of this list,
>
>   Thank you for the quick response. In short, neither of these solutions
> worked for the 16K cpu (1 process per cpu) case.
>
>   Before getting into the details, has anyone successfully executed a
> charm++ job on Stampede's large queue using 1024 nodes? If so, I would be
> grateful to know the (i) build of charm++ and (ii) the charmrun options used
> to launch the job. I've tried a variety of settings and, as a result, I'm
> quickly burning through cpu time.
>
>   Jim: thanks for sharing your home directory with me! I copied over your
> mpiexec script, which appears to give the same errors as the script I've
> been using.
>
>   Bilge: I had previously tried with a
>
>    ./build charm++ net-linux-x86_64 ibverbs --with-production -j8
>
>  charm++ build. I didn't mention this because I had carried out more
> experimentation with charmrun command line options using the vanilla net
> build. However, I've also been unsuccessful launching chamrun with 16K cpus
> using ibverbs. I've now explored a variety of batch/timeout/scalable-start
> options (without success) and have built with both
>
> ./build charm++ verbs-linux-x86_64 --with-production -j8
>
> and
>
> ./build charm++ net-linux-x86_64 ibverbs --with-production -j8
>
> ...which might be equivalent?
>
> I've been able to launch hello-world-type mpi programs on all 1024 nodes, so
> I believe it is possible in principle to do the same with charm++.
>
> I would be grateful for any suggestions. I am currently looking into using a
> nodelist instead of mpiexec. However, as previously noted, I'm burning
> through SUs and would like to focus my remaining SUs on solutions most
> likely to work.
>
> Thanks in advance for any tips!
>
> Best,
> Scott
>
> On Tue, Nov 11, 2014 at 1:51 PM, Bilge Acun <acun2 AT illinois.edu> wrote:
>>

>> You should use Infiniband build of Charm++ on Stampede instead of the
>> regular net build. Your build command should look like:
>>
>> >> ./build charm++ verbs-linux-x86_64 --with-production -j8
>>
>> On 11 November 2014 11:59, Jim Phillips <jim AT ks.uiuc.edu> wrote:
>>>
>>>
>>> Here is my (possibly useless) suggestion:
>>>
>>>
>>> Take a look at /home1/00288/tg455591/NAMD_scripts/runbatch_latest
>>>
>>> I use:   ++scalable-start ++mpiexec ++remote-shell $SCRIPTDIR/mpiexec
>>>
>>> where $SCRIPTDIR/mpiexec looks like:
>>>
>>> #!/bin/csh -f
>>>
>>> # drop -n <N>
>>> shift
>>> shift
>>>
>>> exec /usr/local/bin/ibrun $*
>>>
>>>
>>> It looks like you're doing about the same thing.  I can't say I've every
>>> actually tried 16k processes, though.  At some point we switch to smp.
>>>
>>> You may want to try launching an MPI hello world with ibrun at that scale
>>> just to be sure it works at all.  Good luck.
>>>
>>> Jim
>>>
>>>
>>> On Tue, 11 Nov 2014, Scott Field wrote:
>>>
>>> > Hi,
>>> >
>>> > I am running large jobs (as part of a scaling test) on stampede. My
>>> > complete module list is "TACC-paths, cluster-paths, cluster,
>>> > cmake/2.8.9,
>>> > mvapich2/1.9a2,
>>> > Linux, xalt/0.4.4, TACC, gcc/4.7.1, mkl/13.0.2.146"
>>> >
>>> > and charm++ (the most recent version from git) has been built with
>>> >
>>> >>>> ./build charm++ net-linux-x86_64 --with-production -j8
>>> >
>>> > The scaling tests span 1 node (16 procs) up to 1024 nodes (16384
>>> > procs).
>>> > When I hit 256 nodes charmrun starts reporting problems. Typically I
>>> > execute 4 charmruns in a single sbatch submission. At 256 procs the
>>> > first
>>> > one fails:
>>> >
>>> > TACC: Starting up job 4421563
>>> > TACC: Setting up parallel environment for MVAPICH2+mpispawn.
>>> > TACC: Starting parallel tasks...
>>> > Charmrun> error 4466 attaching to node:
>>> > Timeout waiting for node-program to connect
>>> >
>>> > while the next three succeed. At 16384 procs all 4 charmrun jobs fail
>>> > with
>>> > the same error (although the error number is different). My "base"
>>> > command
>>> > is
>>> >
>>> >>>> ./charmrun ./Evolve1DScalarWave +p16384 ++mpiexec ++remote-shell
>>> > mympiexec
>>> >
>>> > where Evolve1DScalarWave is the executable and mympiexec is
>>> >
>>> > #!/bin/csh
>>> > shift; shift; exec ibrun $*
>>> >
>>> > Finally, I've tried numerous possible combinations of the following
>>> > command
>>> > line options
>>> >
>>> > ++scalable-start
>>> > ++timeout XXX
>>> > ++batch YYY
>>> >
>>> > Where XXX is one of 60, 100, 1000 and YYY is one of 10, 64 and 128.
>>> > None of
>>> > these worked. Using only scalable-start I get a slightly modified error
>>> > message
>>> >
>>> >
>>> > Charmrun> error 93523 attaching to node:
>>> > Error in accept.
>>> >
>>> >
>>> > For all three options enabled and a large timeout I get about 200,000
>>> > lines
>>> > from charmrun showing these same lines over and over (with the numbers
>>> > different):
>>> >
>>> > Charmrun remote shell(127.0.0.1.0)> remote responding...
>>> > Charmrun remote shell(127.0.0.1.0)> starting node-program...
>>> > Charmrun remote shell(127.0.0.1.0)> rsh phase successful.
>>> > Charmrun> Waiting for 16305-th client to connect.
>>> > Charmrun> client 6763 connected (IP=129.114.77.58 data_port=42030)
>>> > Charmrun> adding client 12805: "127.0.0.1", IP:127.0.0.1
>>> >
>>> > until finally the job fails with
>>> >
>>> > [c401-004.stampede.tacc.utexas.edu:mpispawn_1][spawn_processes] Failed
>>> > to
>>> > execvp() 'sfield':  (2)
>>> >
>>> > Sometimes this last message is seen immediately after "TACC: Starting
>>> > parallel tasks...".
>>> >
>>> > I've been able to reproduce this problem with the jacobi2D charm++
>>> > example.
>>> >
>>> > Any help or suggestions would be greatly appreciated!
>>> >
>>> > Best,
>>> > Scott
>>> >
>>> _______________________________________________
>>> charm mailing list
>>> charm AT cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/charm
>>
>>
>>
>>
>> --
>> Bilge Acun
>> PhD Candidate at University of Illinois at Urbana-Champaign
>> Computer Science Department
>
>

--
Esteban Meneses
Research Assistant Professor
Center for Simulation and Modeling
University of Pittsburgh

_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm