Skip to Content.
Sympa Menu

charm - Re: [charm] Scalable creation of chare array elements, passing a different portion of a potentially large array to their constructors

charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] Scalable creation of chare array elements, passing a different portion of a potentially large array to their constructors


Chronological Thread 
  • From: "Kale, Laxmikant V" <kale AT illinois.edu>
  • To: Jozsef Bakosi <jbakosi AT gmail.com>, "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
  • Subject: Re: [charm] Scalable creation of chare array elements, passing a different portion of a potentially large array to their constructors
  • Date: Sat, 31 Oct 2015 20:02:14 +0000
  • Accept-language: en-US

Is the data for each chare array element “arbitrary” (i.e. You get it from input from a file, for example), or is it  a function of its index? I assume its arbitrary (otherwise, each element could construct it itself based on thisIndex). 

Secondly, is the data available already on processor 0? Or is it in a file? If it is in a file, one can use some parallel I/O abstraction (we have ckio library, but will need to check its readiness), and have each element read form a specific offset from the file, assuming they are  of fixed size.

You can also scatter data to a distributed table and have each chare pick up what it needs from the table via an explicit request. Distributed table as an abstraction was removed years ago, but thats because it can be easily implemented via a group: just hash the array index to PE, and that where you store all the key-value pairs that map to it ( <key: index, value: data for index’th array element).  I have a feeling this last one is what you want. The key-value table is a tutorial example (in sdag section, I think). We  can dig that out.

-- 
--- 
Laxmikant (Sanjay) Kale         http://charm.cs.uiuc.edu
Professor, Computer Science     kale AT illinois.edu
201 N. Goodwin Avenue           Ph:  (217) 244-0094
Urbana, IL  61801-2302          

From: Jozsef Bakosi <jbakosi AT gmail.com>
Reply-To: Jozsef Bakosi <jbakosi AT gmail.com>
Date: Saturday, October 31, 2015 at 2:51 PM
To: "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
Subject: [charm] Scalable creation of chare array elements, passing a different portion of a potentially large array to their constructors

Hi folks,

I have a chare array, that takes a vector of integers in its constructor, defined by the following:

In .ci file:

array [1D] Performer {
  entry Performer( const std::vector< std::size_t >& data );
};

In .h file:

class Performer : public CBase_Performer {

  public:
    explicit Performer( const std::vector< std::size_t >& data ) :
      m_data( data ) {}

  private:
     std::vector< std::size_t > m_data;
};

I would like to create a large number of Performer array elements, but passing a different vector of integers to the constructor, depending on the index of the array element, in a scalable way.

My first approach was to declare a

std::vector< std::vector< std::size_t > > alldata

at global scope (and have the Charm++ runtime system distribute it to all PEs), in which the outer vector had the size equal to the number of array elements to be created. (This was global scope because it was a result of an MPI library call, done before initializing the Charm++ portion, and I used the data in global-scope to transfer it to the Charm++ main chare.) Thus when the array constructor was called, inside it, I just peeled off the correct part of the data vector using thisIndex. However, that is problematic, since the global-scope alldata is potentially very large so it is not a good idea to have the runtime system try to broadcast it to all PEs. It is also a waste, since only a small fraction is used by each chare array element created.

Then I did away with declaring alldata as a global-scope object and managed to pass it down through the place where I need to fire up Performer chare array elements, which helps, still not a great solution, but I would be okay with it -- if I could somehow now fire up the Performer arrays (1) in a scalable way, and (2) passing a different part of the alldata array to their constructor. Is this possible?

I know I can do:

CProxy_Performer::ckNew( alldata );

But that sends the potentially very large vector of vectors to all array elements -- very wasteful.

I can also do:

CkArrayOptions opts( nchare );   // nchare = number of desired Performer array elements
CProxy_Performer::ckNew( alldata, opts );

But that, again is not really what I want, unless I can somehow setup CkArrayOptions in a way that allows passing only the right part of alldata to the individual array elements. I've also looked into custom initial placement maps via CkArrayOptions, but that does not seem like what I need here either.

Of course, I can also create the array elements manually, one by one, in a for-loop, but that then does not use the tree-based (bulk) array-element creation Charm++ provides, so I don't think that will scale well, since a potentially very large number of Performer chare array elements will be spawned from a single host chare.

I also thought about doing a two-step array creation by first creating a Charm++ group (i.e., one instance on each PE), and then spawn the Performers from there, resulting in a tree-based object creation, but then I have to deal with PEs, which PE will fire up Performers initialized by which part of alldata, etc. It seems like there should be a better way than that too.

I'm sure you guys must have run into this kind of stuff probably years ago and I'm just missing something or don't know what functionality to use based on the documentation. It seems like what I need is a way to use the built-in bulk array element creation mechanism, but passing a different set of data to the individual array elements. Put in yet another way, I need to know thisIndex before I call ckNew on the array (so I can peel off just the correct small portion of alldata and pass it). How can this be done? Or rather, how should this be done? What I really need is to distribute a very large data (i.e., alldata, potentially mega or gigabytes in memory) to chares and only sending part of it to each.

Thanks,
Jozsef



Archive powered by MHonArc 2.6.16.

Top of Page