Charm++ parallel programming system

["Christian Tuma" <tuma AT zib.de>]

Hello,

I have built NAMD 2.9 (including charm 6.4.0). Our machine has nodes
equipped with one dual-port InfiniBand adaptor each. One port is connected
to our communication network (MPI, interface "ib0"), the other port is
dedicated to I/O (Lustre, interface "ib1"). When running NAMD I observe
that it uses the "wrong" port, that is, NAMD uses our I/O network ("ib1")
for communication.

How can I make NAMD communicate over the other InfiniBand port (using
"ib0")? From an earlier thread on this list in June 2012 I have learnt
that the port number is hard-coded into charm 6.4.0. I took the simple and
naive approach to change line 470 in
charm-6.4.0/src/arch/net/machine-ibverbs.c

from
"ibPort=1;"

to
"ibPort=0;"


but this didn't help, it crashes when doing the pgm test in
charm-6.4.0/tests/charm++/megatest.

What can I do, any suggestions from the experts?


Some more info ...

For building charm I use

./build charm++ net-linux-x86_64 gcc ibverbs gfortran -j8 -O3
-mtune=native -ffast-math

Output of 'ibstat':

CA 'mlx4_0'
        CA type: MT26418
        Number of ports: 2
        Firmware version: 2.7.0
        Hardware version: a0
        Node GUID: 0x0030487fb6180000
        System image GUID: 0x0030487fb6180003
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 20
                Base lid: 149
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0030487fb6180001
                Link layer: IB
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 20
                Base lid: 229
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510868
                Port GUID: 0x0030487fb6180002
                Link layer: IB



Appreciating your help -

Christian