charm AT lists.cs.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] Charm++ performance issue on CSCS PizDaint

From: Matthias Diener <mdiener AT illinois.edu>
To: Bignamini Christopher <christopher.bignamini AT cscs.ch>
Cc: "charm AT lists.cs.illinois.edu" <charm AT lists.cs.illinois.edu>
Subject: Re: [charm] Charm++ performance issue on CSCS PizDaint
Date: Thu, 10 Mar 2022 11:29:06 -0600
Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=illinois.edu; dmarc=pass action=none header.from=illinois.edu; dkim=pass header.d=illinois.edu; arc=none
Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ph8N0rMVLuwv8UcFQQvr/vjTNhWjFMJwcAueNpre1XA=; b=edBehSxqOs664C0q/HnmZQMZ0XWprzTrsP9Hs2A+r277wvqJ1atKmEf5oIQeOseBvnzKFQZm+wZS8t7XnyPcS7O5HD+iDbl6I2HzAxV0+NJ6NYZron91oK+ctmNrvzLni305Iw6USk5Kt4SgO9kWR+NRl+abMIce2ke9m35o8/pZ8izq0lVQI1nezwqoAcSTPqLL3atU/LMJmWZln0BQnSE3aNIQN8r7MWlRAkfNHEX0c7rIDSih3Opx2kbeOSo2umQ17fEPG5NcjGElRSzfiT73DdjZUakxyZ0UGG2i0heStq8BfEo6cZmYUWJcv5n5SSErVbRhvNjtsxduA3TNeA==
Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jXMZYaBahENjh/UyLpETTeM4eXLtr21fFVLLN5r/4vV+ZQmThK18uYuImgwnnvpDw0DGCjyIDJFbGWt3LBqi8s1t+1peADGlSERo5NWLEl5D7T8WJ4hZIFfH4ePRmlTEC7aTK38Si2PTnJzdh7TxytZab2f8mne5jeyPGHT7GOMX8UNwSHQ2T6sMuxyRuhQpAxLITHv1vVTqAt+nH6jigDpwtS30MqW3o/PM+taGK/fwl+ejIWbzOdY5rv2QKOGZQ/om/DIUqNOHYUszO8dkBKWDcqy7STE2VTH1l6wTrGw+xzLWWJYSgxK963LG7IcgQZyWSzH73XW2MpBf9TKIDw==
Authentication-results: ppops.net; spf=softfail smtp.mailfrom=mdiener AT illinois.edu; dkim=pass header.s=selector2-uillinoisedu-onmicrosoft-com header.d=uillinoisedu.onmicrosoft.com; dmarc=fail header.from=illinois.edu

Hi Christopher,

Thank you for your report. Could you please let us know which command
you use to build Charm++, and how you are running NAMD?

We have heard reports of some network performance issues on PizDaint
(unrelated to Charm++), but we are not sure what is causing them.

Best,
Matthias

On Tue, Mar 8, 2022 at 9:45 AM Bignamini Christopher
<christopher.bignamini AT cscs.ch>
wrote:
>
> Hello,
>
>
>
> my name is Christopher Bignamini (CSCS-ETH Zurich) and I’m contacting you
> concerning a recent issue I’m having with NAMD/Charm++ on one of our
> systems.
>
> We have a regression testsuite that we use to measure system performances
> and it includes some CPU/GPU NAMD tests, where we measure performances
>
> of our NAMD/Charm++ installation (Intel based): after a recent upgrade of
> the system (not of the Charm++/NAMD version), performance degradation is
> present
>
> for the CPU only version of NAMD in a 16 nodes run configuration while
> tests running on smaller set of nodes (6) and/or with GPU acceleration
> (same tests) do not show any issue.
>
> By profiling the executable, I found a possible indication that the problem
> I am seeing could related to the interaction between Charm++ and our system
> network,
>
> particularly through the following function:
>
>
>
> https://charm.cs.illinois.edu/doxygen/charm/machine-common-core_8C-source.shtml#l01818
>
>
>
> Results of my profiling runs show a completely different time usage between
> a 6 nodes and a 16 nodes job (percentage refers to sampling number):
>
> 6 nodes: 39.0% CmiGetNonLocalNodeQ
> 16 nodes: 51.6% CmiGetNonLocalNodeQ
>
> with the amount of time spent in USER function (NAMD computation) that is
> quite reduced
>
> 6 nodes: 19.2% USER
> 16 nodes: 9.5% USER
>
> I know that I am only providing general, high level information but I am
> still trying to understand what is the problem and I am a bit lost: do you
> have any suggestion
>
> concerning where I can look in Charm++ code to understand what is reducing
> its performances? Any help is appreciated!
>
>
>
> Thank you very much in advance for your attention, have a nice day!
>
> Cheers
> Christopher

[charm] Charm++ performance issue on CSCS PizDaint, Bignamini Christopher, 03/08/2022
- <Possible follow-up(s)>
- Re: [charm] Charm++ performance issue on CSCS PizDaint, Matthias Diener, 03/10/2022