Charm++ parallel programming system

[Jozsef Bakosi <jbakosi AT lanl.gov>]

Hi folks,

So far we have used Charm++ in non-SMP mode and now we would like to run
our regression tests in SMP mode as well. We have tests with varying
number of PEs and we use cmake/ctest to schedule the tests concurrently
on a single compute node.

So far we only used charmrun's +p command line argument but now we would
like to run the tests in SMP mode so that the tests with more than one
PE do exercise SMP mode.

Most importantly, we would like to preserve the number of work units for
all tests so we don't have to update (and maintain separate,
SMP-specific) baselines.

Some degree of thread-oversubscription is not a big problem, e.g., due
to the dedicated communication threads compared to non-SMP mode, unless
it severely degrades performance.

I tried various SMP, multicore, and +oneWth* options but none seem to be
suitable.

For example, we would like a test with say 4 PEs to run in SMP mode
with, e.g., 2 processes and 2 worker threads, for which I tried the
following:

+p4 +ppn 2, which gives:

Reason: Multiple PEs assigned to same core. Set affinity options to correct 
or lower the number of threads, or pass +setcpuaffinity to ignore.

+p4 +ppn 2 +setcpuaffinity, runs fine but (probably due to
thread-contention) close to an order of magnitude slower compared to

+p4 +ppn 1, which runs fine but does not exercise SMP.

Can someone give a good advice on how we should specify the runtime
options for tests so that SMP is exercised by the parallel tests yet the
overall test suite still does not oversubscribe threads too much?

Thanks,
Jozsef