[jallan@hpc27 ~]$ `which  mpirun` --report-bindings --bind-to-core -mca btl openib,sm,self -np 8   -hostfile ./hplnodes2 -d --debug-daemons  /cm/shared/apps/hpl/2.0/xhpl 
[hpc27:06010] System has detected external process binding to cores 0080
[hpc27:06010] procdir: /tmp/openmpi-sessions-jallan@hpc27_0/50313/0/0
[hpc27:06010] jobdir: /tmp/openmpi-sessions-jallan@hpc27_0/50313/0
[hpc27:06010] top: openmpi-sessions-jallan@hpc27_0
[hpc27:06010] tmp: /tmp
[hpc27:06010] mpirun: reset PATH: /cm/shared/apps/openmpi/gcc/64/1.4.4/bin:/opt/vdt/globus/sbin:/opt/vdt/globus/bin:/opt/vdt/apache-maven/bin:/opt/vdt/apache-ant/bin:/opt/vdt/gpt/sbin:/opt/vdt/jdk1.6/bin:/opt/moab_lambda/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/bin:/sbin:/usr/sbin:/cm/shared/apps/hpl/2.0:/cm/shared/apps/openmpi/gcc/64/1.4.4/bin:/chome/jallan/bin
[hpc27:06010] mpirun: reset LD_LIBRARY_PATH: /cm/shared/apps/openmpi/gcc/64/1.4.4/lib64:/opt/vdt/globus/lib:/opt/vdt/jdk1.6/jre/lib/amd64client:/opt/vdt/jdk1.6/jre/lib/amd64/server:/opt/vdt/jdk1.6/jre/lib/amd64:/cm/shared/apps/gotoblas/core/64/1.26:/cm/shared/apps/openmpi/gcc/64/1.4.4/lib64
aklog: Couldn't determine realm of user:aklog: unknown RPC error (-1765328189)  while getting realm
aklog: Couldn't determine realm of user:aklog: unknown RPC error (-1765328189)  while getting realm
Daemon was launched on hpc23 - beginning to initialize
Daemon was launched on hpc24 - beginning to initialize
[hpc23:22184] procdir: /tmp/openmpi-sessions-jallan@hpc23_0/50313/0/1
[hpc23:22184] jobdir: /tmp/openmpi-sessions-jallan@hpc23_0/50313/0
[hpc23:22184] top: openmpi-sessions-jallan@hpc23_0
[hpc23:22184] tmp: /tmp
[hpc24:09206] procdir: /tmp/openmpi-sessions-jallan@hpc24_0/50313/0/2
[hpc24:09206] jobdir: /tmp/openmpi-sessions-jallan@hpc24_0/50313/0
[hpc24:09206] top: openmpi-sessions-jallan@hpc24_0
[hpc24:09206] tmp: /tmp
Daemon [[50313,0],2] checking in as pid 9206 on host hpc24
Daemon [[50313,0],2] not using static ports
[hpc24:09206] [[50313,0],2] orted: up and running - waiting for commands!
Daemon [[50313,0],1] checking in as pid 22184 on host hpc23
Daemon [[50313,0],1] not using static ports
[hpc27:06010] [[50313,0],0] node[0].name hpc27 daemon 0 arch ffc91200
[hpc27:06010] [[50313,0],0] node[1].name hpc23 daemon 1 arch ffc91200
[hpc27:06010] [[50313,0],0] node[2].name hpc24 daemon 2 arch ffc91200
[hpc27:06010] [[50313,0],0] node[3].name hpc12 daemon INVALID arch ffc91200
[hpc27:06010] [[50313,0],0] orted_cmd: received add_local_procs
[hpc23:22184] [[50313,0],1] orted: up and running - waiting for commands!
[hpc23:22184] [[50313,0],1] node[0].name hpc27 daemon 0 arch ffc91200
[hpc23:22184] [[50313,0],1] node[1].name hpc23 daemon 1 arch ffc91200
[hpc23:22184] [[50313,0],1] node[2].name hpc24 daemon 2 arch ffc91200
[hpc23:22184] [[50313,0],1] node[3].name hpc12 daemon INVALID arch ffc91200
[hpc23:22184] [[50313,0],1] orted_cmd: received add_local_procs
[hpc24:09206] [[50313,0],2] node[0].name hpc27 daemon 0 arch ffc91200
[hpc24:09206] [[50313,0],2] node[1].name hpc23 daemon 1 arch ffc91200
[hpc24:09206] [[50313,0],2] node[2].name hpc24 daemon 2 arch ffc91200
[hpc24:09206] [[50313,0],2] node[3].name hpc12 daemon INVALID arch ffc91200
[hpc24:09206] [[50313,0],2] orted_cmd: received add_local_procs
--------------------------------------------------------------------------
Not enough processors were found on the local host to meet the requested
binding action:

  Local host:        hpc23
  Action requested:  bind-to-core
  Application name:  /cm/shared/apps/hpl/2.0/xhpl

Please revise the request and try again.
--------------------------------------------------------------------------
[hpc23:22184] [[50313,0],1] orted_cmd: received iof_complete cmd
--------------------------------------------------------------------------
mpirun was unable to start the specified application as it encountered an error:

Error name: Unknown error: 1
Node: hpc23.ib.ptc

when attempting to start process rank 0.
--------------------------------------------------------------------------
[hpc27:06010] [[50313,0],0] orted_cmd: received exit
/cm/shared/apps/hpl/2.0/xhpl: error while loading shared libraries: libgoto.so: cannot open shared object file: No such file or directory
[hpc23:22184] [[50313,0],1] orted_cmd: received iof_complete cmd
[hpc23:22184] [[50313,0],1] orted_cmd: received exit
[hpc23:22184] [[50313,0],1] orted: finalizing
[hpc27:06010] sess_dir_finalize: job session dir not empty - leaving
[hpc27:06010] 1 more process has sent help message help-odls-default.txt / odls-default:not-enough-resources
[hpc27:06010] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[hpc27:06010] sess_dir_finalize: proc session dir not empty - leaving
orterun: exiting with status 1
[jallan@hpc27 ~]$ [hpc23:22184] sess_dir_finalize: job session dir not empty - leaving
[hpc23:22184] sess_dir_finalize: job session dir not empty - leaving
[hpc24:09206] [[50313,0],2] orted_cmd: received iof_complete cmd
[hpc24:09206] [[50313,0],2] orted_cmd: received exit
[hpc24:09206] [[50313,0],2] orted: finalizing
[hpc24:09206] sess_dir_finalize: job session dir not empty - leaving
[hpc24:09206] sess_dir_finalize: job session dir not empty - leaving