Charm++ parallel programming system

[Esteban Meneses-Rojas <emeneses AT pitt.edu>]

   Scott,
   I am currently running some 16K-core (1024-node) jobs on Stampede
with Charm++.
   Here is the build I am using:
./build charm++ mpi-linux-x86_64
   And, this is how I run it:
ibrun ./pgm

On Wed, Nov 12, 2014 at 7:58 PM, Scott Field 
<sfield AT astro.cornell.edu>
 wrote:
> Hi Jim, Bilge and the rest of this list,
>
>   Thank you for the quick response. In short, neither of these solutions
> worked for the 16K cpu (1 process per cpu) case.
>
>   Before getting into the details, has anyone successfully executed a
> charm++ job on Stampede's large queue using 1024 nodes? If so, I would be
> grateful to know the (i) build of charm++ and (ii) the charmrun options used
> to launch the job. I've tried a variety of settings and, as a result, I'm
> quickly burning through cpu time.
>
>   Jim: thanks for sharing your home directory with me! I copied over your
> mpiexec script, which appears to give the same errors as the script I've
> been using.
>
>   Bilge: I had previously tried with a
>
>    ./build charm++ net-linux-x86_64 ibverbs --with-production -j8
>
>  charm++ build. I didn't mention this because I had carried out more
> experimentation with charmrun command line options using the vanilla net
> build. However, I've also been unsuccessful launching chamrun with 16K cpus
> using ibverbs. I've now explored a variety of batch/timeout/scalable-start
> options (without success) and have built with both
>
> ./build charm++ verbs-linux-x86_64 --with-production -j8
>
> and
>
> ./build charm++ net-linux-x86_64 ibverbs --with-production -j8
>
> ...which might be equivalent?
>
> I've been able to launch hello-world-type mpi programs on all 1024 nodes, so
> I believe it is possible in principle to do the same with charm++.
>
> I would be grateful for any suggestions. I am currently looking into using a
> nodelist instead of mpiexec. However, as previously noted, I'm burning
> through SUs and would like to focus my remaining SUs on solutions most
> likely to work.
>
> Thanks in advance for any tips!
>
> Best,
> Scott
>
> On Tue, Nov 11, 2014 at 1:51 PM, Bilge Acun 
> <acun2 AT illinois.edu>
>  wrote:
>>
>> You should use Infiniband build of Charm++ on Stampede instead of the
>> regular net build. Your build command should look like:
>>
>> >> ./build charm++ verbs-linux-x86_64 --with-production -j8
>>
>> On 11 November 2014 11:59, Jim Phillips 
>> <jim AT ks.uiuc.edu>
>>  wrote:
>>>
>>>
>>> Here is my (possibly useless) suggestion:
>>>
>>>
>>> Take a look at /home1/00288/tg455591/NAMD_scripts/runbatch_latest
>>>
>>> I use:   ++scalable-start ++mpiexec ++remote-shell $SCRIPTDIR/mpiexec
>>>
>>> where $SCRIPTDIR/mpiexec looks like:
>>>
>>> #!/bin/csh -f
>>>
>>> # drop -n <N>
>>> shift
>>> shift
>>>
>>> exec /usr/local/bin/ibrun $*
>>>
>>>
>>> It looks like you're doing about the same thing.  I can't say I've every
>>> actually tried 16k processes, though.  At some point we switch to smp.
>>>
>>> You may want to try launching an MPI hello world with ibrun at that scale
>>> just to be sure it works at all.  Good luck.
>>>
>>> Jim
>>>
>>>
>>> On Tue, 11 Nov 2014, Scott Field wrote:
>>>
>>> > Hi,
>>> >
>>> > I am running large jobs (as part of a scaling test) on stampede. My
>>> > complete module list is "TACC-paths, cluster-paths, cluster,
>>> > cmake/2.8.9,
>>> > mvapich2/1.9a2,
>>> > Linux, xalt/0.4.4, TACC, gcc/4.7.1, mkl/13.0.2.146"
>>> >
>>> > and charm++ (the most recent version from git) has been built with
>>> >
>>> >>>> ./build charm++ net-linux-x86_64 --with-production -j8
>>> >
>>> > The scaling tests span 1 node (16 procs) up to 1024 nodes (16384
>>> > procs).
>>> > When I hit 256 nodes charmrun starts reporting problems. Typically I
>>> > execute 4 charmruns in a single sbatch submission. At 256 procs the
>>> > first
>>> > one fails:
>>> >
>>> > TACC: Starting up job 4421563
>>> > TACC: Setting up parallel environment for MVAPICH2+mpispawn.
>>> > TACC: Starting parallel tasks...
>>> > Charmrun> error 4466 attaching to node:
>>> > Timeout waiting for node-program to connect
>>> >
>>> > while the next three succeed. At 16384 procs all 4 charmrun jobs fail
>>> > with
>>> > the same error (although the error number is different). My "base"
>>> > command
>>> > is
>>> >
>>> >>>> ./charmrun ./Evolve1DScalarWave +p16384 ++mpiexec ++remote-shell
>>> > mympiexec
>>> >
>>> > where Evolve1DScalarWave is the executable and mympiexec is
>>> >
>>> > #!/bin/csh
>>> > shift; shift; exec ibrun $*
>>> >
>>> > Finally, I've tried numerous possible combinations of the following
>>> > command
>>> > line options
>>> >
>>> > ++scalable-start
>>> > ++timeout XXX
>>> > ++batch YYY
>>> >
>>> > Where XXX is one of 60, 100, 1000 and YYY is one of 10, 64 and 128.
>>> > None of
>>> > these worked. Using only scalable-start I get a slightly modified error
>>> > message
>>> >
>>> >
>>> > Charmrun> error 93523 attaching to node:
>>> > Error in accept.
>>> >
>>> >
>>> > For all three options enabled and a large timeout I get about 200,000
>>> > lines
>>> > from charmrun showing these same lines over and over (with the numbers
>>> > different):
>>> >
>>> > Charmrun remote shell(127.0.0.1.0)> remote responding...
>>> > Charmrun remote shell(127.0.0.1.0)> starting node-program...
>>> > Charmrun remote shell(127.0.0.1.0)> rsh phase successful.
>>> > Charmrun> Waiting for 16305-th client to connect.
>>> > Charmrun> client 6763 connected (IP=129.114.77.58 data_port=42030)
>>> > Charmrun> adding client 12805: "127.0.0.1", IP:127.0.0.1
>>> >
>>> > until finally the job fails with
>>> >
>>> > [c401-004.stampede.tacc.utexas.edu:mpispawn_1][spawn_processes] Failed
>>> > to
>>> > execvp() 'sfield':  (2)
>>> >
>>> > Sometimes this last message is seen immediately after "TACC: Starting
>>> > parallel tasks...".
>>> >
>>> > I've been able to reproduce this problem with the jacobi2D charm++
>>> > example.
>>> >
>>> > Any help or suggestions would be greatly appreciated!
>>> >
>>> > Best,
>>> > Scott
>>> >
>>> _______________________________________________
>>> charm mailing list
>>> charm AT cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/charm
>>
>>
>>
>>
>> --
>> Bilge Acun
>> PhD Candidate at University of Illinois at Urbana-Champaign
>> Computer Science Department
>
>



-- 
Esteban Meneses
Research Assistant Professor
Center for Simulation and Modeling
University of Pittsburgh