Charm++ parallel programming system

["Kale, Laxmikant V" <kale AT illinois.edu>]

I think we should take this discussion off the general Charm++ mailing list. So, the next message on this should omit charm AT cs.illinois.edu (to avoid clutter for the rest). The following is the result of my discussion with Phil Miller. 

Full C++ parsing, since our early attempts at it, was considered not worth trying.  Charm++ does not try to parse C++ code at all. The “.ci” file was designed to be something that we can easily parse. Thats the sole reason for the limitation. 

We're sorry for this limitation. We've avoided implementing full _expression_ parsing in the .ci file parser because of the complexity of doing that in a general manner that doesn't leave arbitrary restrictions that would be even less comprehensible than "charmxi can't parse expressions".

To address the initial use case that put this limitation in your path, we're going to implement support for multi-dimensional arrays shortly (i.e. it should be in the mainline of our repository this week).

In the longer term, we're going to look at using libclang to do the _expression_ and declaration parsing for us, with full generality of the language.

BTW: in your opinion, in the interim, is it better to have support for some simple expressions including your examples, but not the full generality of C++ expressions?

I think, in the meanwhile, using a node-group to maintain this, is a solution I’d recommend. The groups are all initialized before any chare array elements start running (you have to call the group creation before the array creation so the ordering is preserved on PE 0 as well), and getting the size via a public member and local pointer is as efficient. Or use readonly messages (section 3.3 of the manual ), which allow dynamically sized read-only data structures; you have to write (simple, in this case) pack/unpack methods for the message. 

We will review this issue in the next weekly meetings of Charm++, and come out with a statement of our plan for the mailing list.

— Sanjay

From: "<Van Der Wijngaart>", Rob F <rob.f.van.der.wijngaart AT intel.com>
Date: Monday, September 29, 2014 at 3:33 PM
To: "Kunzman, David M" <david.m.kunzman AT intel.com>, "Hammond, Jeff R" <jeff.r.hammond AT intel.com>, Phil Miller <mille121 AT illinois.edu>
Cc: Nikhil Jain <nikhil.life AT gmail.com>, "Mattson, Timothy G" <timothy.g.mattson AT intel.com>, "charm AT cs.illinois.edu" <charm AT cs.illinois.edu>
Subject: Re: [charm] Global variables in Charm++

Thanks, Dave.

 

Actually, as I wrote last week, I wanted to be able to define something a little more complex, which turned out to be totally unpalatable by charmc:

readonly double weight[2*RADIUS+1][2*RADIUS+1];

Why would that be a reasonable thing to do for a readonly variable, one may ask? The reason is that this is an array that specifies the weights of a stencil. In my workload I allow different stencil sizes and different weights. But once defined, weight is a global constant. I let the user specify the constant symbol RADIUS, and use that to compute weight’s dimensions. In the .C file I can do that to my heart’s content, but not in the .ci file. To me that is not reasonable. I think this points to a critical weakness of Charm++. The only things that get supported in the processing of .ci files are:

1.       Constructs that are foreseen by the Charm++ developers.

2.       Constructs that the Charm++ developers deem to have merit.

 

Rob

 

From: Kunzman, David M
Sent: Monday, September 29, 2014 1:20 PM
To: Van Der Wijngaart, Rob F; Hammond, Jeff R; Phil Miller
Cc: Nikhil Jain; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: RE: Global variables in Charm++

 

In principle, I don’t see a reason why what you are trying to do (using an ‘_expression_’ in the array dimension) could not be supported in ci files.  I think this is just a case of, charmxi (the tool that parses and processes the ci files) not being a “full blown” compiler.  Taking a quick look at the grammar for charmxi, you’ll see…

 

Readonly        : READONLY Type QualName DimList

 

DimList         : /* Empty */

                | Dim DimList

 

Dim             : '[' ArrayDim ']'

 

ArrayDim        : NUMBER

                | QualName

 

QualName        : IDENT

                | QualName ':'':' IDENT

 

… So, within the square brackets, you either need a number or a qualified name.  Rob, in your case, I think the preprocessor is leaving an _expression_ after replacing RADIUS, which just isn’t currently defined in the charmxi grammar.

 

That said, to be honest with you, I don’t really recall people using readonly variables in the way that you are using them (data for the calculation itself), except in very simple test/example codes, for two reasons…

 

(1)  Essentially, you are programming in a style that tries to use “globals” in an object oriented programming model (nothing inherently wrong with this, but people tend to avoid globals in object oriented programming models such as C++, which is unlike MPI where people rely on globals more often or in C).  This is not to say that people don’t do this (they obviously do, because quite frankly, it gets the job done in many cases).

 

(2)  In Charm++, readonly also have the restriction that they must have their value set after the main chare constructors are finished executing, so they tend to be somewhat limited in usefulness for anything beyond the simplest of codes.  In production codes, application startup tends to occur after of the main chare constructors have completed, so using something like a group or a node group to calculate and/or transmit runtime constants (lookup tables, weight matrices, whatever) makes more sense from a Charm++ perspective.

 

Because of item 2 in particular, I don’t think people have really cared about this in the past before.

 

Thanks,

Dave K

 

 

 

From: Van Der Wijngaart, Rob F
Sent: Monday, September 29, 2014 12:49 PM
To: Hammond, Jeff R; Phil Miller
Cc: Nikhil Jain; Kunzman, David M; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: RE: Global variables in Charm++

 

But then again, maybe “double weight[4.5+4.5];” is OK. In some parallel universe this must be true.

 

Rob

 

From: Van Der Wijngaart, Rob F
Sent: Monday, September 29, 2014 12:47 PM
To: Hammond, Jeff R; Phil Miller
Cc: Nikhil Jain; Kunzman, David M; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: RE: Global variables in Charm++

 

Eh, that would stink spectacularly. I refuse to accept that “double weight[3+3+3];” may result in a syntax error. I invite the Charm++ team to explain why this would be reasonable.

 

Rob

 

From: Hammond, Jeff R
Sent: Monday, September 29, 2014 12:38 PM
To: Van Der Wijngaart, Rob F; Phil Miller
Cc: Nikhil Jain; Kunzman, David M; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: Re: Global variables in Charm++

 

Can you do something evil where you extern pointer to storage and then declare it in a generic C++ file that will be preprocessed properly?

 

Obviously, may provoke the question “what is that terrible smell?” but I’ve seen far worse workarounds associated with legacy Fortran :-)

 

Jeff

 

From: "<Van Der Wijngaart>", Rob F <rob.f.van.der.wijngaart AT intel.com>
Date: Monday, September 29, 2014 at 9:55 AM
To: Phil Miller <mille121 AT illinois.edu>
Cc: Nikhil Jain <nikhil.life AT gmail.com>, "Kunzman, David M" <david.m.kunzman AT intel.com>, Jeff R Hammond <jeff.r.hammond AT intel.com>, "Mattson, Timothy G" <timothy.g.mattson AT intel.com>, "charm AT cs.illinois.edu" <charm AT cs.illinois.edu>
Subject: RE: Global variables in Charm++

 

OK, it appears not to be the problem of running the preprocessor per se, but of having expressions in array dimensions. I replaced

  readonly double weight[(2*RADIUS+1)*(2*RADIUS+1)];

with

  readonly double weight[3+3+3];

and got another syntax error.

This looks like a pretty serious limitation to me. I am now required to do the array dimension computation for charmc and stuff that into the .ci file explicitly as a constant. This means running another script/preprocessor before I can run charmc.

 

Rob

 

From: Van Der Wijngaart, Rob F
Sent: Friday, September 26, 2014 3:49 PM
To: 'Phil Miller'
Cc: Nikhil Jain; Kunzman, David M; Hammond, Jeff R; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: RE: Global variables in Charm++

 

Thanks Phil. I just added –E to my OPTS parameter, but then all hell breaks loose.

This is what charmc says is happening when it tries to compile my .ci file:

../../../bin/charmc -Ofast -DRADIUS=1 -DSTAR -E  jacobi2d.ci

And this is the line in my .ci file that it complains about:

  readonly double weight[(2*RADIUS+1)*(2*RADIUS+1)];

 

Output:

[rfvander@bar1 PRKstencil5]$ make

../../../bin/charmc -Ofast -DRADIUS=1 -DSTAR -E  jacobi2d.ci

STDIN:8: Charmxi syntax error> syntax error

Invalid construct

Fatal Error by charmc in directory /home/rfvander/charm/examples/charm++/PRKstencil5

   Command ../../../bin/charmxi -orig-file jacobi2d.ci returned error code 1

charmc exiting...

../../../bin/charmc -Ofast -DRADIUS=1 -DSTAR -c jacobi2d.C

jacobi2d.C(1): catastrophic error: cannot open source file "jacobi2d.decl.h"

  #include "jacobi2d.decl.h"

 

By the way, you may wonder about the subject line of this message. It started because I wanted a (small) 2D array as a global variable. I couldn’t get that to work (is that by design?), so made it into a 1D array. And then I ran into the issue I just reported.

 

Rob

 

From:unmobile AT gmail.com [mailto:unmobile AT gmail.com] On Behalf Of Phil Miller
Sent: Friday, September 26, 2014 3:37 PM
To: Van Der Wijngaart, Rob F
Cc: Nikhil Jain; Kunzman, David M; Hammond, Jeff R; Mattson, Timothy G; charm AT cs.illinois.edu
Subject: Re: Global variables in Charm++

 

For historical reasons [1], charmc defaults to *not* running the preprocessor on .ci files. Preprocessing .ci files can be enabled by passing charmc the -E flag when it's working on .ci files.

[1] Enabling pre-processing when it wasn't historically the default risks breaking existing application code by having macros that may have been used as identifiers in .ci files suddenly be subject to expansion/substitution.

 

On Fri, Sep 26, 2014 at 5:31 PM, Van Der Wijngaart, Rob F <rob.f.van.der.wijngaart AT intel.com> wrote:

Hello Nikhil,

 

I have a symbol that I define on the compile line (i.e. I stuff it in OPTS, as below):

OPTS=-DRADIUS=1

CHARMC=../../../bin/charmc $(OPTS)

When I now use RADIUS in my .C file, the C preprocessor does its work nicely and recognizes that I have defined RADIUS, and also what its value is. However, when I use RADIUS in the .ci file, charmc croaks. So is it true that no preprocessor is run on .ci files? Thanks!

 

Rob