Charm++ parallel programming system

["Kale, Laxmikant V" <kale AT illinois.edu>]

There is a fixed amount of memory allocated to storing traces (which can be controlled, within limits, by +logsize parameter. (to be used while running the application). 

Once this runs out, on any given processor (separately), that processor will flush the log to the log file before continuing. If these flushes (which take milliseconds) happen at different times on different processors, the performance data will be distorted (ompared with an execution that is not recording traces)  because of these interruptions. Thats what the warning is telling you.

The trace should not be corrupted by this (if it is, thats another problem). Just that it will show long gaps on individual processors during flushing. 

Synchronizing: use application logic to find a point at which to flush traces from all processors, via an explicit call. (will someone post the name of the call, and update the manual to list in the above-linked appendix E?).

Another option, for fine grained applications that generate a lot of events, is to only sample a portion of the execution in detail (turning traces on and off explicitly; see “Selective tracing” in appendix E of the manual), and optionally, to trace the entire execution in summary mode.

Hope this helps. (and not too obvious a summary for you).

-- 

--- 

Laxmikant (Sanjay) Kale         http://charm.cs.uiuc.edu

Professor, Computer Science     kale AT illinois.edu

201 N. Goodwin Avenue           Ph:  (217) 244-0094

Urbana, IL  61801-2302          

From: Alexander Frolov <alexndr.frolov AT gmail.com>
Date: Wednesday, July 22, 2015 at 6:11 AM
To: "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
Subject: [charm] Synchronizing flushing of collected traces to log files

Hi!

I am profiling my charm++ application with projections. It works well until I increase the problem size and receive the warnings like that:

Warning: Projections log flushed to disk 4 times on 2 cores: 0 1 .

Warning: The performance data is likely invalid, unless the flushes have been explicitly synchronized by your program. 

In this case the trace is corrupted. How can I synchronize it?

Best, 

   Alex