Charm++ parallel programming system

["Sheppard, Raymond W" <rsheppar AT iu.edu>]

Hi again,
  I had not heard anything about my offer to sponsor accounts, so I figured I 
would send a few compile attempts from different paths I have taken.  The 
word is that they will try brining up a few GPU racks soon.  For now though, 
this is a CPU only system.  Please let me know if anyone in your group would 
like access.  
         Ray


rsheppar@login1:/N/soft/sles15/namd/build/namd_july_2021./charm-6.10.2> 
./build                   charm++ mpi-linux-x86_64 craycc  --with-production
Selected Compiler: craycc
Selected Options:
Creating dir: mpi-linux-x86_64-craycc
Creating dir: mpi-linux-x86_64-craycc/tmp
Soft-linking over bin
Soft-linking over lib
Soft-linking over include
Soft-linking over tmp
Copying src/scripts/Makefile to mpi-linux-x86_64-craycc/tmp
Generating mpi-linux-x86_64-craycc/tmp/conv-mach-pre.sh
Generating mpi-linux-x86_64-craycc/tmp/conv-mach-opt.h, conv-mach-opt.sh, 
conv-m                  ach-opt.mak
Performing '/usr/bin/gmake basics OPTS=-optimize -production QUIET= 
CONFIG_OPTS=                  --disable-controlpoint --disable-tracing 
--disable-tracing-commthread --disable-                  charmdebug 
--disable-replay --disable-error-checking --disable-stats ' in mpi-li         
         nux-x86_64-craycc/tmp
pwd
/geode2/soft/hps/sles15/namd/build/namd_july_2021./charm-6.10.2/mpi-linux-x86_64
                  -craycc/tmp
find . -type l -exec rm {} \;
rm -rf QuickThreads
rm -rf hwloc
rm -rf libs
rm -rf ../bin ; mkdir ../bin
if case mpi-linux-x86_64 in *win64*) true;; *-win-*) true;; *) false;; esac; 
the                  n \
        cp -f ../../src/arch/win/unix2nt* ../bin; \
        cp -f ../../src/arch/win/createlink.exe ../bin; \
fi
rm -rf ../lib ; mkdir ../lib
rm -rf ../benchmarks
rm -rf ../examples
rm -rf ../tests
../../src/scripts/gatherflat ../../src/scripts          .
./gatherflat ../../src/conv-core                        .
./gatherflat ../../src/conv-ldb                         .
./gatherflat ../../src/conv-ccs                         .
./gatherflat ../../src/conv-perf                        .
./gatherflat ../../src/ck-core                          .
./gatherflat ../../src/ck-perf                          .
./gatherflat ../../src/ck-pics                          .
./gatherflat ../../src/ck-ldb                           .
./gatherflat ../../src/ck-cp                            .
./gatherflat ../../src/langs/simplemsg                  .
./gatherflat ../../src/langs/pvmc                       .
./gatherflat ../../src/langs/bluegene                   .
./gatherflat ../../src/langs/f90charm                   .
./gatherflat ../../src/xlatcpm                          .
./gathertree ../../src/QuickThreads                     QuickThreads
./gathertree ../../src/../contrib/hwloc                 hwloc
./gathertree ../../src/libs                             libs
./gathertree ../../src/arch/util                        .
./gathertree ../../src/arch/util/proc_management        proc_management
./gathertree ../../src/util                             .
./gathertree ../../src/langs                            langs
./gathertree ../../src/langs/jade                       langs/jade
./gathertree ../../src/arch/common                      .
./gathertree ../../src/arch/mpi                 .
test -f ../../src/arch/mpi/gdir_link && cat ../../src/arch/mpi/gdir_link > 
.gdir.new && ./gathertree ../../src/arch/`cat .gdir.new`      . || true
./gatherflat ../../src/arch/mpi-linux-x86_64                 .
./gathertree ../../src/../benchmarks                    ../benchmarks
./gathertree ../../src/../examples                      ../examples
./gathertree ../../src/../tests                         ../tests
rm -f ../bin/dep.pl ; cp dep.pl ../bin/
chmod +x charmc
./system_ln  ../tmp/charmc ../bin/
./system_ln  ../../src/scripts/testrun ../bin/
./system_ln  ../tmp/tree.txt ../bin/
./system_ln  ../tmp/fuzzytree.txt ../bin/
rm -rf ../include ; mkdir ../include
./system_ln  ../tmp/conv-*.*h ../include
./system_ln  ../tmp/cc-*.*h ../include
./system_ln  ../tmp/conv-mach-opt.sh ../include
./system_ln  ../tmp/conv-mach-opt.mak ../include
if [ -x ./special.sh ] ; then SRCBASE=../../src ./special.sh ; fi
cc is not CrayCC, check your modules!
gmake: *** [Makefile:412: dirs+sources] Error 1
-------------------------------------------------
Charm++ NOT BUILT. Either cd into mpi-linux-x86_64-craycc/tmp and try
to resolve the problems yourself, visit
http://charm.cs.illinois.edu/
for more information. Otherwise, email the developers at 
charm AT cs.illinois.edu
rsheppar@login1:/N/soft/sles15/namd/build/namd_july_2021./charm-6.10.2> 
module list

Currently Loaded Modules:
  1) cce/11.0.4              5) craype-network-ofi       9) 
cray-libsci/21.05.1.1
  2) craype/2.7.7            6) cray-dsmml/0.1.5        10) PrgEnv-cray/8.0.0
  3) craype-x86-rome/1.0.0   7) perftools-base/21.05.0
  4) libfabric/1.11.0.3.71   8) cray-mpich/8.1.5
_________________________________________________________________________________


rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> export 
MPICXX=CC
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> export 
CXX=CC
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> export 
CC=cc
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> export 
F90=ftn
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> export 
F77=ftn
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> ./build 
charm++ mpi-linux-x86_64 mpicxx --with-production
Selected Compiler: mpicxx
Selected Options:
Creating dir: mpi-linux-x86_64-mpicxx
Creating dir: mpi-linux-x86_64-mpicxx/tmp
Soft-linking over bin
Soft-linking over lib
Soft-linking over include
Soft-linking over tmp
Copying src/scripts/Makefile to mpi-linux-x86_64-mpicxx/tmp
Generating mpi-linux-x86_64-mpicxx/tmp/conv-mach-pre.sh
Generating mpi-linux-x86_64-mpicxx/tmp/conv-mach-opt.h, conv-mach-opt.sh, 
conv-mach-opt.mak
Performing '/usr/bin/gmake basics OPTS=-optimize -production QUIET= 
CONFIG_OPTS=--disable-controlpoint --disable-tracing 
--disable-tracing-commthread --disable-charmdebug --disable-replay 
--disable-error-checking --disable-stats ' in mpi-linux-x86_64-mpicxx/tmp
pwd
/geode2/soft/hps/sles15/namd/build/namd_7_2021/charm-6.10.2/mpi-linux-x86_64-mpicxx/tmp
find . -type l -exec rm {} \;
rm -rf QuickThreads
rm -rf hwloc
rm -rf libs
rm -rf ../bin ; mkdir ../bin
if case mpi-linux-x86_64 in *win64*) true;; *-win-*) true;; *) false;; esac; 
then \
        cp -f ../../src/arch/win/unix2nt* ../bin; \
        cp -f ../../src/arch/win/createlink.exe ../bin; \
fi
rm -rf ../lib ; mkdir ../lib
rm -rf ../benchmarks
rm -rf ../examples
rm -rf ../tests
../../src/scripts/gatherflat ../../src/scripts          .
./gatherflat ../../src/conv-core                        .
./gatherflat ../../src/conv-ldb                         .
./gatherflat ../../src/conv-ccs                         .
./gatherflat ../../src/conv-perf                        .
./gatherflat ../../src/ck-core                          .
./gatherflat ../../src/ck-perf                          .
./gatherflat ../../src/ck-pics                          .
./gatherflat ../../src/ck-ldb                           .
./gatherflat ../../src/ck-cp                            .
./gatherflat ../../src/langs/simplemsg                  .
./gatherflat ../../src/langs/pvmc                       .
./gatherflat ../../src/langs/bluegene                   .
./gatherflat ../../src/langs/f90charm                   .
./gatherflat ../../src/xlatcpm                          .
./gathertree ../../src/QuickThreads                     QuickThreads
./gathertree ../../src/../contrib/hwloc                 hwloc
./gathertree ../../src/libs                             libs
./gathertree ../../src/arch/util                        .
./gathertree ../../src/arch/util/proc_management        proc_management
./gathertree ../../src/util                             .
./gathertree ../../src/langs                            langs
./gathertree ../../src/langs/jade                       langs/jade
./gathertree ../../src/arch/common                      .
./gathertree ../../src/arch/mpi                 .
test -f ../../src/arch/mpi/gdir_link && cat ../../src/arch/mpi/gdir_link > 
.gdir.new && ./gathertree ../../src/arch/`cat .gdir.new`      . || true
./gatherflat ../../src/arch/mpi-linux-x86_64                 .
./gathertree ../../src/../benchmarks                    ../benchmarks
./gathertree ../../src/../examples                      ../examples
./gathertree ../../src/../tests                         ../tests
rm -f ../bin/dep.pl ; cp dep.pl ../bin/
chmod +x charmc
./system_ln  ../tmp/charmc ../bin/
./system_ln  ../../src/scripts/testrun ../bin/
./system_ln  ../tmp/tree.txt ../bin/
./system_ln  ../tmp/fuzzytree.txt ../bin/
rm -rf ../include ; mkdir ../include
./system_ln  ../tmp/conv-*.*h ../include
./system_ln  ../tmp/cc-*.*h ../include
./system_ln  ../tmp/conv-mach-opt.sh ../include
./system_ln  ../tmp/conv-mach-opt.mak ../include
if [ -x ./special.sh ] ; then SRCBASE=../../src ./special.sh ; fi
if [ ! -f conv-common.h ] ; then ( touch conv-common.h ) ; fi
touch dirs+sources
/usr/bin/gmake -C topomanager -f Makefile_charm headers
gmake[1]: Entering directory 
'/geode2/soft/hps/sles15/namd/build/namd_7_2021/charm-6.10.2/mpi-linux-x86_64-mpicxx/tmp/topomanager'
/bin/cp TopoManager.h ../../include
/bin/cp BGQTorus.h ../../include
/bin/cp XTTorus.h ../../include
echo "//empty" > topomanager_config.h
/bin/cp topomanager_config.h ../../include
gmake[1]: Leaving directory 
'/geode2/soft/hps/sles15/namd/build/namd_7_2021/charm-6.10.2/mpi-linux-x86_64-mpicxx/tmp/topomanager'
/usr/bin/gmake charmxi
gmake[1]: Entering directory 
'/geode2/soft/hps/sles15/namd/build/namd_7_2021/charm-6.10.2/mpi-linux-x86_64-mpicxx/tmp'
./configure --disable-controlpoint --disable-tracing 
--disable-tracing-commthread --disable-charmdebug --disable-replay 
--disable-error-checking --disable-stats
Charm++/LIBS error checking is disabled
AMPI error checking is enabled
Statistics collection is disabled
Charm tracing is disabled
Charm tracing communication thread is disabled
Task Queue is disabled
Drone mode is disabled
CharmDebug is disabled
Charm record/replay is disabled
CCS is enabled
Charm control point is disabled
Charm LB user data is disabled
Lockless queue for PE local and node queue is disabled
Charm shrink expand is disabled
charm4py support is disabled
Setting load balancing timer type as 'double'
checking machine name... mpi-linux-x86_64-mpicxx
checking "cp command as"... cp -p
checking "C++ compiler as"... "CC    "
checking "whether C++ compiler works"... "ok"
checking "C++ linker as"... "CC     "
checking "whether linker works"... "ok"
checking "Native C++ compiler as"... "  "
checking "Sequential C++ compiler as"... "CC  "
checking "whether compiler accept -fno-stack-protector"... "ok"
checking "whether C++ compiler supports C++11 without flags"... "yes"
checking "whether native C++ compiler supports C++11 without flags"... "no"
checking "whether C++ compiler supports C++11 with '-std=c++11'"... "no"
checking "whether C++ compiler supports C++11 with '--c++11'"... "no"
checking "whether C++ compiler supports C++11 with '-h std=c++11'"... "no"
checking "whether C++ compiler accepts -fno-lifetime-dse"... "no"
checking "whether C++ compiler accepts -mno-tls-direct-seg-refs"... "yes"
Setting charm++ envelope refnum field to unsigned short
Configuring support for message priorities of arbitrary size (bitvectors)
Using mempool cutoff... 2^26
checking "whether compiler generates code for 64-bit"... "yes"
checking "whether compiler supports std::is_constructible"... "yes"
checking "whether compiler supports std::alignment_of"... "yes"
checking "whether compiler implements regex"... "yes"
checking "whether has values.h "... "yes"
checking "whether has stdint.h "... "yes"
checking "whether has malloc.h "... "yes"
checking "whether has alloca.h "... "yes"
checking "whether has regex.h "... "yes"
checking "whether long long works"... "yes"
checking "whether __int64 works"... "no"
checking "whether __int128 (128-bit integer) works"... "yes"
checking "whether __int128_t (128-bit integer) works"... "yes"
checking "whether C++ library has <atomic> "... "yes"
checking "whether long double works"... "yes"
checking "whether ucontext has FPU pointer"... "yes"
checking "whether ucontext uses uc_regs"... "no"
checking "whether ucontext has pointer (v_regs) of vector type"... "no"
checking "whether ibverbs ibv_port_attr has link_layer field"... "yes"
checking "whether inline works in C"... "no"
checking "whether C++ signed char and char differ"... "yes"
checking "whether typeinfo/typeid works"... "ok"
checking "whether std::iterator_traits is defined"... "ok"
checking "whether std::distance is defined"... "ok"
checking "whether std::inserter is defined"... "ok"
checking "whether std::void_t is defined"... "ok"
checking "whether operator delete can be overloaded in same class"... "ok"
checking "whether offsetof is defined"... "yes"
checking "whether GCC x86 assembly works"... "yes"
checking "whether GCC x86 assembly for atomic increment works"... "yes"
checking "whether asm eieio assembly works"... "no"
checking "whether __thread (Thread Local Storage) is supported"... "yes"
checking "whether synchronization primitives (__sync_add_and_fetch) works in 
C"... "no"
checking "whether synchronization primitives (__sync_synchronize) works in 
C"... "no"
checking "whether has __executable_start"... "no"
checking "whether switching TLS register (64-bit) is supported"... "yes"
checking "whether has dl_iterate_phdr"... "no"
checking "whether has __malloc_hook"... "no"
checking "whether build on MPI"... "no"
Error: Cannot compile an MPI program
*** Please find detailed output in tmp/charmconfig.out ***
gmake[1]: *** No rule to make target 'conv-autoconfig.h', needed by 
'xi-main.o'.  Stop.
gmake[1]: Leaving directory 
'/geode2/soft/hps/sles15/namd/build/namd_7_2021/charm-6.10.2/mpi-linux-x86_64-mpicxx/tmp'
gmake: *** [Makefile:331: headers] Error 2
-------------------------------------------------
Charm++ NOT BUILT. Either cd into mpi-linux-x86_64-mpicxx/tmp and try
to resolve the problems yourself, visit
http://charm.cs.illinois.edu/
for more information. Otherwise, email the developers at 
charm AT cs.illinois.edu
rsheppar@login1:/N/soft/sles15/namd/build/namd_7_2021/charm-6.10.2> module 
list

Currently Loaded Modules:
  1) aocc/2.2.0.1            5) craype-network-ofi       9) 
cray-libsci/21.05.1.1
  2) craype/2.7.7            6) cray-dsmml/0.1.5        10) PrgEnv-aocc/8.0.0
  3) craype-x86-rome/1.0.0   7) perftools-base/21.05.0
  4) libfabric/1.11.0.3.71   8) cray-mpich/8.1.5


________________________________________
From: Bohm, Eric J 
<ebohm AT illinois.edu>
Sent: Thursday, July 29, 2021 3:56 PM
To: Sheppard, Raymond W
Cc: 
charm AT cs.illinois.edu
Subject: [External] Re: Need help building on a hybrid system.

This message was sent from a non-IU address. Please exercise caution when 
clicking links or opening attachments from external sources.

Hello,

Could you provide some of the critical output from the unsuccessful 
compilation attempts?

We're currently working on getting access to that architecture so we can 
provide more useful assistance.
________________________________
From: Sheppard, Raymond W 
<rsheppar AT iu.edu>
Sent: Saturday, July 24, 2021 1:59 AM
To: 
charm AT cs.illinois.edu
 
<charm AT cs.illinois.edu>
Subject: Re: [charm] Need help building on a hybrid system.

Hello again,
  Just a follow up of things I omitted that may be helpful.  The system is 
running SLES-15 and my module list looks like this:

rsheppar@login1:/N/soft/sles15/namd/build/namd_july_2021./charm-6.10.2/src/arch>
 module list

Currently Loaded Modules:
  1) aocc/2.2.0.1   3) craype-x86-rome/1.0.0   5) craype-network-ofi   7) 
perftools-base/21.05.0   9) cray-libsci/21.05.1.1
  2) craype/2.7.7   4) libfabric/1.11.0.3.71   6) cray-dsmml/0.1.5     8) 
cray-mpich/8.1.5        10) PrgEnv-aocc/8.0.0


Thanks in advance for any help building here.
        Ray

________________________________________
From: Sheppard, Raymond W 
<rsheppar AT iu.edu>
Sent: Friday, July 23, 2021 12:21 PM
To: 
charm AT cs.illinois.edu
Subject: Need help building on a hybrid system.

Hi,
  I am lost.  We are building a new Cray (HPE) system.  Software wise, it 
looks like a Cray (PrgEnv-accc, cray-mpich is preferred).
 However it is made of those new AMD chips (128 cores on a node in co-located 
pairs of 64 with a shared memory access penalty between pairs).  There is no 
GNI. OpenFabrics is the default and UDP can be sort of made to work.  I think 
I need something in between:

./build charm++ mpi-linux-x86_64 mpicxx --with-production

and

./build charm++ mpi-linux-x86_64 craycc  --with-production

I've tried a lot of other crazy stuff (like pretending it is an XC40, 
renaming compiler env vars, etc.), but no luck yet.  I keep thinking I can 
pull pieces out of one file to fill in another, but I have not hit the magic 
combination.  Any help appreciated.  Thanks.
  Ray