Charm++ parallel programming system

[Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>]

Thank you very much for your answer!

No, I don't plan to use offload mode. The main goal is to  attempt to experiment in an multi-node environment with Intel Xeon Haswells and Intel Knights Corner CPUs, where Intel Knight Corner are used in native mode. So I do need MPI to communicate between nodes.

If you can provide some information about your machine, we can give you more pointed advice on how best to execute in your environment.

I would be very grateful! Thank you for your time!

Best regards,

Ekaterina

2016-10-04 1:00 GMT+03:00 Ronak Buch <rabuch2 AT illinois.edu>:

What sort of heterogeneous environment do you want to execute in?  I assume since you're talking about MKL that you're interested in using Xeon Phis.  If so, MKL should take care of the heterogeneous execution (see https://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf for details).

As for the compiler, Charm++ doesn't really have its own compiler, we just wrap another compiler under the hood.  Presumably in this case you'll want to use icc, which should handle the AVX2 vectorization.

One other thing is that you seem to have built an MPI build of Charm++.  In general, building on top of MPI isn't the best way to build Charm++, as it is often slower than our own native layers.  If you can provide some information about your machine, we can give you more pointed advice on how best to execute in your environment.

Thanks,

Ronak

On Mon, Oct 3, 2016 at 4:50 PM, Mikida, Eric P <mikida2 AT illinois.edu> wrote:

Ah, that is my fault. I was compiling with an older version of Charm++. The comlib module was deprecated, and is no longer included in Charm 6.7.1. I’ve pushed another fix that removes the -module comlib from Makefile so it should build fine now. For your second question, if it can’t find ckmulticast.h it probably means you didn’t build LIBS. Did you accidentally clean the build before building everylb?

I’m not as familiar with the heterogeneous configuration part, so I’ll make sure to hand that off to someone else in the lab.

Thanks

Eric

On Oct 3, 2016, at 7:54 AM, Ekaterina Tutlyaeva <xgl AT rsc-tech.ru> wrote:

Dear Eric,

Thank you for your fix!

Now compilation phase passes successfully, but I've encountered another issue during linking stage. It requires some "module comlib":

Generating dependencies for driver.ci
Generating dependencies for lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o lu.o lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o scheduler.o scheduler.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o benchmark.o benchmark.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -module comlib -module CkMulticast  -o charmlu lu.o scheduler.o benchmark.o -L/opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a /opt/intel/mkl/lib/intel64/libmkl_core.a /opt/intel/mkl/lib/intel64/libmkl_sequential.a /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl
/usr/bin/ld: cannot find -lmodulecomlib
collect2: error: ld returned 1 exit status
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu

My CHARM is builded with the next options:

./build LIBS mpi-linux-x86_64 mpicxx

and I can't found the module comlib; there also no tests with it. I have charm-6.7.1.

What am I doing wrong now?

I'm tried to compile EvelyLB target for charm:

./build everylb mpi-linux-x86_64 mpicxx

where mpicxx is Intel mpi.

But the dense LU benchmark again encountered compilation issue, now with the ckmulticast.h header file for this Charm build:

Generating dependencies for driver.ci
Generating dependencies for lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o lu.o lu.C
In file included from manager.h:5:0,
                 from lu.h:1,
                 from lu.C:13:
messages.h:5:25: fatal error: ckmulticast.h: No such file or directory
 #include <ckmulticast.h>
                         ^
compilation terminated.

Could you, please, give me advise, how the CHARM should be configured to use the heterogeneous environment, Intel MPI (and what about avx2 features for the compiler)?
I want to try to run the dense LU benchmark in heterogeneous environment with Intel CPUs...

Thank you very much for your time!

best regards,

Ekaterina

2016-10-03 3:40 GMT+03:00 Mikida, Eric P <mikida2 AT illinois.edu>:

Hi Ekaterina,

This one was actually our fault. There was a syntax error in lu.ci that popped up after a small change to how the SDAG language is defined. I fixed that error and pushed my change back to the repository. If you do a pull and try to compile again, it should work this time but let me know if more problems crop up.

Thanks

Eric

On Oct 2, 2016, at 10:03 AM, Ekaterina Tutlyaeva <xgl AT rsc-tech.ru> wrote:

Dear Sir/Madam,

I'm trying to compile the dense LU benchmark: http://charmplusplus.org/miniApps/#denselu

and I've encountered problems in linking it with Intel MKL BLAS.

It's my first experience with CHARM, so sorry for the maybe obvious question.

I have next error:

Generating dependencies for driver.ci
Generating dependencies for lu.ci
STDIN:303:575-575: error: invalid SDAG member
  };
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ^
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
   Command /storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmxi -M -orig-file lu.ci returned error code 1
charmc exiting...
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
In file included from benchmark.C:2:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
 #error "No BLAS Header files included!"
  ^
Generating dependencies for scheduler.C
Generating dependencies for lu.C
In file included from lu.C:15:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
 #error "No BLAS Header files included!"
  ^
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51  -c -o lu.o lu.C
In file included from lu.h:1:0,
                 from lu.C:13:
manager.h:4:26: fatal error: luUtils.decl.h: No such file or directory
 #include "luUtils.decl.h"
                          ^
compilation terminated.
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
   Command mpicxx -m64 -fPIC -I/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/../include -D__CHARMC__=1 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51 -O3 -fno-stack-protector -std=c++0x -c lu.C -o lu.o returned error code 1
charmc exiting...
make: *** [lu.o] Error 1

My config.mk contains:

SEND_LIM  = 2
MKLROOT = /opt/intel/mkl
BLAS_INC =  -I${MKLROOT}/include
BLAS_LIBS = ${MKLROOT}/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl

The include path do contains BLAS headers:

ls -la /opt/intel/mkl/include/mkl_blas.h
-rw-r--r--. 1 root root 76919 Jun 29 14:25 /opt/intel/mkl/include/mkl_blas.h

 

 

Please, could you be so kind to give me some hint, what am I doing wrong? Where should I start?

Should I define MKL usage somewhere?

 Thank you for your time!

 

__________

Best regards,

Ekaterina Tyutlyaeva

--

__________

С уважением,

Тютляева Екатерина

--

__________

С уважением,

Тютляева Екатерина