Charm++ parallel programming system

["Galvez Garcia, Juan Jose" <jjgalvez AT illinois.edu>]

Hi Chris,

 

For Cray XE6, TopoManager considers each Gemini router to be a physical node (so the 2 nodes connected to the same router have the same 3D coordinate). The size of the 4-th coordinate is the total number of processors in both nodes, and the T coordinate should uniquely identify a core inside a physical node. But I think the reported number for T dimension may depend on the ppn requested for the job, so if you request ppn=16, T=32, if ppn=32, T=64.

 

-Juan

 

From: Chris Wailes
Sent: Tuesday, February 13, 2018 10:53 AM
To: charm
Subject: [charm] TopoManager / Cray XE6 Question

 

This documentation states that the TopoManager method `getDimNT()`:

Returns the length of T dimension. TopoManager uses T dimension to represent different cores that reside within a physical node.

However, the Cray documentation for the XE6 series of machines states that each Gemini router is connected to two nodes, each of which contains two processors.

My question then is if the X, Y, and Z coordinates identify routers, are all of the cores on all CPUs for both nodes connected to a router represented using the T dimension or are there other, hidden, dimensions that specify the Cray node and processors?

I ask because the XE6 machine I'm currently using (which has 16-core, 2-way SMP-enabled CPUs) is reporting X, Y, Z, and T dimensions of 11, 6, 8, and 32.  This would seem to indicate that a coordinate quadruple doesn't uniquely identify a core, but instead identifies four cores, one on each CPU attached to each node attached to a router.

Any clarification would be appreciated, and thanks for your work on Charm++.

- Chris