Charm++ parallel programming system

[Phil Miller <mille121 AT illinois.edu>]

Ooh, I see. I forgot you were on an MPI build of Charm++, rather than the IP-based network layer (net-* or netlrts-*) we normally use in development. The charmrun utility is different for each network layer, and I think only the net-* version of charmrun supports that option. On other layers, since it's not responsible for process launching, it has nothing to report.

The other flags I mentioned should be more helpful, though.

On Fri, Sep 12, 2014 at 12:49 PM, Van Der Wijngaart, Rob F <rob.f.van.der.wijngaart AT intel.com> wrote:

OK, Phil. An interesting thing happens when I do that. Charmrun expects the ++verbose parameter to follow the name of the program to be executed. If I place it before that name (which is where one would normally expect it to be), the program name is not found and charmrun aborts. OK, so I run the program like this.

./charmrun +p4 ./jacobi2d 2000 ++verbose

Inside the application the number of command line parameters is determined, and if it is not two (i.e. program name plus optional grid size), the default grid size is used. Adding the ++verbose flag produces three command line parameters, and the default grid size is used. So you can be verbose, or change the grid size, but not both. Of course, I can easily hack around this in the app. But I would recommend changing the charmrun command line parsing and put all non-app parameters before the name of the app. That is what users expect, and also would make charmrun conform to mpirun

 

Anyway, here is the output:

 

[rfvander@bar1 jacobi2d]$ ./charmrun +p4 ./jacobi2d 2000 ++verbose

Running on 4 processors:  ./jacobi2d 2000 ++verbose

charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 4  ./jacobi2d 2000 ++verbose

Charm++> Running on MPI version: 3.0

Charm++> level of thread support used: MPI_THREAD_SINGLE (desired: MPI_THREAD_SINGLE)

Charm++> Running in non-SMP mode: numPes 4

Converse/Charm++ Commit ID:

Trace: traceroot: ./jacobi2d

CharmLB> Load balancer assumes all CPUs are same.

Charm++> Running on 1 unique compute nodes (32-way SMP).

Charm++> cpu topology info is gathered in 0.003 seconds.

Running Jacobi on 4 processors with (5,5) elements; myproc=0.0

 

Rob

 

From: unmobile AT gmail.com [mailto:unmobile AT gmail.com] On Behalf Of Phil Miller
Sent: Friday, September 12, 2014 10:35 AM

To: Van Der Wijngaart, Rob F
Cc: charm AT cs.illinois.edu
Subject: Re: [charm] Problem building charm++ on Intel platform

 

Hi Rob,

Could you add the flag "++verbose" to your command line and show us the output? That will show how and where charmrun launches processes.

If this is all on one node, it may be necessary to pass a flag like "+setcpuaffinity +pemap 0-3" to pin the PEs to separate cores. It would be very unusual that the OS scheduler doesn't run the multiple processes on separate cores, though, even if they may occasionally shuffle around.

Details on how these flags work can be found in our manual, here: http://charm.cs.illinois.edu/manuals/html/charm++/C.html

Phil

 

On Fri, Sep 12, 2014 at 11:53 AM, Van Der Wijngaart, Rob F <rob.f.van.der.wijngaart AT intel.com> wrote:

Hi Phil,

 

I am merrily playing around with the charm++ examples before creating my own applications. Right now I am looking at the jacobi2d code. I’ve been trying to make that run on more than a single core. But no matter how I run it, it never wants to do that. For example, if I run it like this:

./charmrun +p4 jacobi2d 2000

it works on a 2000x2000 grid (in a very fine-grain manner, of course, so not very efficient), and it still insists that only core 0 on node zero is used. Is there a way to coerce/tickle that runtime to use more cores? Do I need to make the code coarser grain fist? Thanks.

 

Rob

 

From: unmobile AT gmail.com [mailto:unmobile AT gmail.com] On Behalf Of Phil Miller
Sent: Tuesday, September 02, 2014 3:54 PM

To: Van Der Wijngaart, Rob F
Cc: charm AT cs.illinois.edu
Subject: Re: [charm] Problem building charm++ on Intel platform

 

On Tue, Sep 2, 2014 at 5:49 PM, Van Der Wijngaart, Rob F <rob.f.van.der.wijngaart AT intel.com> wrote:

Hi Phil, another quick question: If I install Charm++ in /tmp on a node in my cluster, it won’t be visible on other nodes. Is that going to bite me? Thanks.

 

The default build process generates static binaries and library archives (libfoo.a) of the various components of Charm++. These components, and applications linked against them, should have no runtime dependence on the build tree.

When running the code, you may need to copy the application executable itself and the 'charmrun' binary to a parallel filesystem visible to the compute nodes and the head node, respectively.

 

 

Rob

 

From: unmobile AT gmail.com [mailto:unmobile AT gmail.com] On Behalf Of Phil Miller
Sent: Tuesday, September 02, 2014 2:59 PM
To: Van Der Wijngaart, Rob F
Cc: charm AT cs.illinois.edu
Subject: Re: [charm] Problem building charm++ on Intel platform

 

Could you try your build on a local filesystem (perhaps /tmp) or one hosted on NFS rather than Lustre?

 

On Tue, Sep 2, 2014 at 3:00 PM, Van Der Wijngaart, Rob F <rob.f.van.der.wijngaart AT intel.com> wrote:

Dear Charm++ developers,

 

I tried to build Charm++ on a standard Intel Xeon platform with the following command:

./build charm++ mpi-linux-x86_64 ifort mpicxx

Ifort and mpicxx are both installed, and on my path. However, the build always fails in the following way (complete output follows; I’m highlighting the error).

Can you tell me how to resolve this issue? Thanks!

 

Regards,

Rob

 

find . -type l -exec rm {} \;

rm -rf QuickThreads

rm -rf libs

rm -rf ../bin ; mkdir ../bin

rm -rf ../lib ; mkdir ../lib

rm -rf ../lib_so ; mkdir ../lib_so; touch ../lib_so/.charmso

rm -rf ../examples

rm -rf ../tests

rm -rf ../doc ; ln -s ../doc ../doc

../../src/scripts/gatherflat ../../src/scripts .

./gatherflat ../../src/conv-core        .

./gatherflat ../../src/conv-ldb         .

./gatherflat ../../src/conv-ccs         .

./gatherflat ../../src/conv-perf        .

./gatherflat ../../src/ck-core          .

./gatherflat ../../src/util             .

./gatherflat ../../src/ck-perf          .

./gatherflat ../../src/ck-ldb           .

./gatherflat ../../src/ck-com           .

./gatherflat ../../src/ck-cp            .

./gatherflat ../../src/conv-com         .

./gatherflat ../../src/langs/simplemsg  .

./gatherflat ../../src/langs/pvmc       .

./gatherflat ../../src/langs/bluegene   .

./gatherflat ../../src/langs/f90charm   .

./gatherflat ../../src/xlat-i           .

./gatherflat ../../src/xlatcpm          .

./gathertree ../../src/QuickThreads     QuickThreads

./gathertree ../../src/libs             libs

./gathertree ../../src/arch/util        .

./gathertree ../../src/langs            langs

./gathertree ../../src/langs/jade       langs/jade

./gathertree ../../src/arch/common      .

./gathertree ../../src/arch/`cat .gdir` .

test -f ../../src/arch/`cat .gdir`/gdir_link && cat ../../src/arch/`cat .gdir`/gdir_link > .gdir.new && ./gathertree ../../src/arch/`cat .gdir.new`      . || true

./gatherflat ../../src/arch/`cat .vdir` .

./gathertree ../../examples             ../examples

./gathertree ../../tests                ../tests

rm -f ../bin/dep.pl ; cp dep.pl ../bin/

chmod +x charmc

./system_ln  ../tmp/charmc ../bin/

rm -rf ../include ; mkdir ../include

./system_ln  ../tmp/conv-*.*h ../include

./system_ln  ../tmp/cc-*.*h ../include

./system_ln  ../tmp/conv-mach-opt.sh ../include

if [ -x ./special.sh ] ; then ./special.sh ; fi

if [ ! -f conv-common.h ] ; then ( touch conv-common.h ) ; fi

touch dirs+sources

/usr/bin/gmake charmxi

gmake[1]: Entering directory `/lustre/home/rfvander/charm-6.5.1/mpi-linux-x86_64-ifort-mpicxx/tmp'

Makefile:1144: Variable OPTS is defined to an empty string. Are you sure this is what you want?

./configure

Error checking is enabled

Statistics collection is enabled

Charm tracing is enabled

Charm tracing communication thread is disabled

CharmDebug is enabled

Charm record/replay is enabled

CCS is enabled

Charm control point is enabled

Setting load balancing timer type as 'double'

checking machine name... mpi-linux-x86_64-ifort-mpicxx

checking "cp command as"... cp -p

checking "C++ compiler as"... "mpicxx -m64 -fPIC      "

checking "whether C++ compiler works"... "ok"

checking "C++ linker as"... "mpicxx -m64 -fPIC      "

checking "whether linker works"... "ok"

checking "Native C++ compiler as"... "g++ -m64 -fPIC "

checking "Sequential C++ compiler as"... "mpicxx -m64 -fPIC "

checking "whether compiler accept -fno-stack-protector"... "ok"

Setting charm++ envelope refnum field to unsigned short

Configuring support for message priorities of arbitrary size (bitvectors)

checking "whether compiler generates code for 64-bit"... "yes"

checking "whether has strings.h "... "yes"

checking "whether has values.h "... "yes"

checking "whether has stdint.h "... "yes"

checking "whether has malloc.h "... "yes"

checking "whether has alloca.h "... "yes"

checking "whether has regex.h "... "yes"

checking "whether C++ bool works"... "ok"

checking "whether long long works"... "yes"

checking "whether __int64 works"... "no"

checking "whether __int128 (128-bit integer) works"... "yes"

checking "whether __int128_t (128-bit integer) works"... "yes"

checking "whether long double works"... "yes"

checking "whether ucontext has FPU pointer"... "yes"

checking "whether ucontext uses uc_regs"... "no"

checking "whether ucontext has pointer (v_regs) of vector type"... "no"

checking "whether ibverbs ibv_port_attr has link_layer field"... "yes"

checking "whether inline works in C"... "yes"

checking "whether C++ class explicit keyword works"... "ok"

checking "whether C++ signed char and char differ"... "yes"

checking "whether C++ *_casts<> work"... "ok"

checking "whether C++ allows declaration of varsize array"... "yes"

checking "whether namespaces work"... "ok"

checking "whether typeinfo/typeid works"... "ok"

checking "whether std::iterator_traits is defined"... "ok"

checking "whether std::distance is defined"... "ok"

checking "whether std::inserter is defined"... "ok"

checking "whether std::unordered_map is defined"... "no"

checking "whether anon structs are permitted"... "yes"

checking "whether operator delete can be overloaded in same class"... "ok"

checking "whether offsetof is defined"... "yes"

checking "whether GCC x86 assembly works"... "yes"

checking "whether GCC x86 assembly for atomic increment works"... "yes"

checking "whether GCC IA64 assembly works"... "no"

checking "whether asm eieio assembly works"... "no"

checking "whether __thread (Thread Local Storage) is supported"... "yes"

checking "whether synchronization primitives (__sync_add_and_fetch) works in C"... "yes"

checking "whether synchronization primitives (__sync_synchronize) works in C"... "yes"

checking "whether fence intrinsic primitives (__builtin_Xfence_ia32) works in C"... "no"

checking "whether switching TLS register (64-bit) is supported"... "yes"

checking "whether build on MPI"... "yes"

checking "whether need to specify MPI library"... "no"

checking "whether MPI_Init_thread is supported"... "yes"

checking "whether getrusage accepts RUSAGE_THREAD"... "yes"

checking "whether has asctime"... "yes"

checking "whether has log2"... "yes"

checking "whether has sqrtf"... "yes"

checking "whether has fabsf"... "yes"

checking "whether has mkstemp"... "yes"

checking "whether has system"... "yes"

checking "whether has sync()"... "yes"

checking "whether has sbrk"... "yes"

checking "whether has _setjmp/_longjmp"... "yes"

checking "whether has mstats"... "no"

checking "whether has mallinfo"... "yes"

checking "whether has popen"... "yes"

checking "whether has poll"... "yes"

checking "whether has getpagesize"... "yes"

checking "whether has getpid"... "yes"

checking "whether has kill"... "yes"

checking "whether has setpriority"... "yes"

checking "whether to use signal-safe system() "... "no"

checking "whether sched_setaffinity call exists"... "yes"

checking "whether pthread_setaffinity_np call exists"... "yes"

checking "whether pthread_spin_lock exists"... "yes"

checking "whether bindprocessor call exists"... "no"

checking "whether dlopen links without -ldl"... "yes"

checking "whether dlopen links with -ldl"... "yes"

checking "whether gethostname call exists"... "yes"

checking "whether getProcAddress works"... "no"

checking "whether has socklen_t"... "yes"

checking "whether getifaddrs call exists"... "yes"

checking "whether the mmap() syscall exists"... "yes"

checking "whether mmap() accepts MAP_ANON"... "yes"

checking "whether mmap() accepts MAP_NORESERVE"... "yes"

checking "whether has get_myaddress"... "yes"

checking "whether has mprotect"... "yes"

checking "whether glibc backtrace works"... "yes"

checking "whether has sleep "... "yes"

checking "whether has usleep "... "yes"

checking "whether personality() and ADDR_NO_RANDOMIZE exist"... "yes"

checking "whether has zlib"... "yes"

checking "whether has elf.h "... "yes"

checking "whether has Multiprocessing.h for Apple "... "no"

checking "whether ntohl is available"... "yes"

checking "whether has libjpeg"... "no"

checking "whether Python is installed"... "yes"

checking "whether can build shared library"... "yes"

checking "whether can build shared library with MPI"... "yes"

checking for sync... sync

checking "F77 compiler as"... "ifort -auto -fpic "

checking "whether Fortran 77 compiler works"... "yes"

checking "F90 compiler as"... "ifort -auto -fpic "

checking "whether Fortran 90 compiler works"... "yes"

checking subroutine name used by Fortran 90 compiler... ONESCORE

checking Fortran 90 mod name is capital... "no"

checking Fortran 90 mod name extension... "mod"

configure: creating ./config.status

config.status: creating libs/ck-libs/ampi/ampiCC

config.status: creating libs/ck-libs/ampi/ampirun

config.status: creating conv-autoconfig.h

config.status: executing default commands

../bin/charmc -host  xi-main.C

gmake[1]: execvp: ../bin/charmc: Text file busy

gmake[1]: *** [xi-main.o] Error 127

gmake[1]: Leaving directory `/lustre/home/rfvander/charm-6.5.1/mpi-linux-x86_64-ifort-mpicxx/tmp'

gmake: *** [headers] Error 2

-------------------------------------------------

Charm++ NOT BUILT. Either cd into mpi-linux-x86_64-ifort-mpicxx/tmp and try

to resolve the problems yourself, visit

http://charm.cs.illinois.edu/

for more information. Otherwise, email the developers at charm AT cs.illinois.edu

_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm