Charm++ parallel programming system

[Phil Miller <mille121 AT illinois.edu>]

On Wed, May 5, 2010 at 03:22, Haowei Huang 
<huangh AT in.tum.de>
 wrote:
> I mean that , for example, in a program there is a statement
> "Array(x1, y1, x2, y2).insert((currPE++) % numPes)"
>  so all the elements in Array will execute the insert() function like
>  Array(0, 0, 0, 0).insert((currPE++) % numPes);
>  Array(0, 0, 0, 1).insert((currPE++) % numPes);
>  Array(0, 0, 1, 0).insert((currPE++) % numPes);
> ....
>   How to guarantee the sequence of incrementing the value of currPE? For
> example, currPE starts from 0, which element in the array could firstly
> increment the currPE and which one could be the second?  In programmer's
> point of view, they probably want the first element of the array set currPE
> to 1 and the second set the value to 2 and map the elements in the array to
> processor in a round-robin way.

Is this sequence of calls to Array(x1,y1,x2,y2).insert() happening in
a loop somewhere? At this level, Charm++ is just executing sequential
C++ code. In this case, the expression ((currPE++) % numPes) is
evaluated where the call appears. That expression results in currPE's
value increasing by one. The expression's value is then passed to a
function that creates the referenced array element on the PE number
passed. There's no reference to the expression ((currPE++) % numPes)
floating around. The only code accessing currPE is the code making
this call.

On a more general note, if you're creating a dense array whose
elements take no arguments to create, you'd be much better off
creating a map object representing the round-robin arrangement and
creating the array all at once. This is described in the manual:
http://charm.cs.illinois.edu/manuals/html/charm++/3_8.html#SECTION00038400000000000000

>>      3. Could you please tell me a little bit  more about the thread
>> migration to achieve load balance on distributed machine like Cray XT
>> series? Or do you have some documentation describing details of thread
>> migration?
>
>     They don't really explain it. My question is that how does the thread
> migration happen on Cray XT4/XT5
>  for example, within a chip, within a node and between nodes. Could you tell
> just a little bit details or just the
>  overview of it? Then I can check more information by myself.

If you're interested in load balancing, see the manual section about
it: http://charm.cs.illinois.edu/manuals/html/charm++/3_11.html
Threads are not the only things that can migrate in Charm++, so we
don't talk about load balancing in terms of threads, but simply
"migratable objects".

If you're actually interested in the mechanics of thread migration,
the following describes it:

The runtime [0] calls any registered pre-migration functions, creates
a message, packs all data associated with the thread into that message
[1], and sends that message to the destination processor. The runtime
on the destination processor reverses this process to recreate the
thread. None of this is architecture specific. It also doesn't vary by
how close the source and destination PEs are, except that message
delivery to processors in the same node is cheaper than those across
the network.

[0] Migration code:
http://charm.cs.illinois.edu/cgi-bin/gitweb2.cgi?p=charm.git;a=blob;f=src/ck-core/cklocation.C;h=ead8314e301ef02534e348e5cbb07288a8d7227c;hb=HEAD#l2411
[1] Underlying packing code:
http://charm.cs.illinois.edu/cgi-bin/gitweb2.cgi?p=charm.git;a=blob;f=src/libs/ck-libs/tcharm/tcharm.C;h=8975487f1b220e8f1da7f03b428735d4eebd3df2;hb=HEAD#l206

Please address any further follow-ups to the mailing list (CCed) so
that other users can benefit from the discussion.