Hmmm - try adding a value for nprocs instead of leaving it blank. Say “-np 7”
Sent from my iPhone
> On Nov 1, 2018, at 11:56 AM, Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>
> Hello Ralph,
>
> Here is the output for a failing machine:
>
> [130_02:44:13_aleblanc@farbauti]{~}$ > mpirun --mca
> btl_openib_warn_no_device_params_found 0 --mca orte_base_help_aggregate 0
> --mca btl openib,vader,self --mca pml ob1 --mca btl_openib_receive_queues
> P,65536,120,64,32 -hostfile /home/soesterreich/ce-mpi-hosts --mca
> ras_base_verbose 5 IMB-MPI1
>
> ====================== ALLOCATED NODES ======================
> farbauti: flags=0x11 slots=1 max_slots=0 slots_inuse=0 state=UP
> hyperion-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> io-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> jarnsaxa-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> rhea-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> tarqeq-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> tarvos-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> =================================================================
> --------------------------------------------------------------------------
> There are not enough slots available in the system to satisfy the 7 slots
> that were requested by the application:
> 10
>
> Either request fewer slots for your application, or make more slots available
> for use.
> --------------------------------------------------------------------------
>
>
> Here is an output of a passing machine:
>
> [1_02:54:26_aleblanc@hyperion]{~}$ > mpirun --mca
> btl_openib_warn_no_device_params_found 0 --mca orte_base_help_aggregate 0
> --mca btl openib,vader,self --mca pml ob1 --mca btl_openib_receive_queues
> P,65536,120,64,32 -hostfile /home/soesterreich/ce-mpi-hosts --mca
> ras_base_verbose 5 IMB-MPI1
>
> ====================== ALLOCATED NODES ======================
> hyperion: flags=0x11 slots=1 max_slots=0 slots_inuse=0 state=UP
> farbauti-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> io-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> jarnsaxa-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> rhea-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> tarqeq-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> tarvos-ce: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> =================================================================
>
>
> Yes the hostfile is available on all nodes through an NFS mount for all of
> our home directories.
>
>> On Thu, Nov 1, 2018 at 2:44 PM Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>>
>>
>> ---------- Forwarded message ---------
>> From: Ralph H Castain <r...@open-mpi.org>
>> Date: Thu, Nov 1, 2018 at 2:34 PM
>> Subject: Re: [OMPI users] Bug with Open-MPI Processor Count
>> To: Open MPI Users <users@lists.open-mpi.org>
>>
>>
>> I’m a little under the weather and so will only be able to help a bit at a
>> time. However, a couple of things to check:
>>
>> * add -mca ras_base_verbose 5 to the cmd line to see what mpirun thought the
>> allocation was
>>
>> * is the hostfile available on every node?
>>
>> Ralph
>>
>>> On Nov 1, 2018, at 10:55 AM, Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>>>
>>> Hello Ralph,
>>>
>>> Attached below is the verbose output for a failing machine and a passing
>>> machine.
>>>
>>> Thanks,
>>> Adam LeBlanc
>>>
>>>> On Thu, Nov 1, 2018 at 1:41 PM Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>>>>
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Ralph H Castain <r...@open-mpi.org>
>>>> Date: Thu, Nov 1, 2018 at 1:07 PM
>>>> Subject: Re: [OMPI users] Bug with Open-MPI Processor Count
>>>> To: Open MPI Users <users@lists.open-mpi.org>
>>>>
>>>>
>>>> Set rmaps_base_verbose=10 for debugging output
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On Nov 1, 2018, at 9:31 AM, Adam LeBlanc <alebl...@iol.unh.edu> wrote:
>>>>>
>>>>> The version by the way for Open-MPI is 3.1.2.
>>>>>
>>>>> -Adam LeBlanc
>>>>>
>>>>>> On Thu, Nov 1, 2018 at 12:05 PM Adam LeBlanc <alebl...@iol.unh.edu>
>>>>>> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I am an employee of the UNH InterOperability Lab, and we are in the
>>>>>> process of testing OFED-4.17-RC1 for the OpenFabrics Alliance. We have
>>>>>> purchased some new hardware that has one processor, and noticed an issue
>>>>>> when running mpi jobs on nodes that do not have similar processor
>>>>>> counts. If we launch the MPI job from a node that has 2 processors, it
>>>>>> will fail and stating there are not enough resources and will not start
>>>>>> the run, like so:
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> There are not enough slots available in the system to satisfy the 14
>>>>>> slots
>>>>>> that were requested by the application:
>>>>>> IMB-MPI1
>>>>>>
>>>>>> Either request fewer slots for your application, or make more slots
>>>>>> available
>>>>>> for use.
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> If we launch the MPI job from the node with one processor, without
>>>>>> changing the mpirun command at all, it runs as expected.
>>>>>>
>>>>>> Here is the command being run:
>>>>>>
>>>>>> mpirun --mca btl_openib_warn_no_device_params_found 0 --mca
>>>>>> orte_base_help_aggregate 0 --mca btl openib,vader,self --mca pml ob1
>>>>>> --mca btl_openib_receive_queues P,65536,120,64,32 -hostfile
>>>>>> /home/soesterreich/ce-mpi-hosts IMB-MPI1
>>>>>>
>>>>>> Here is the hostfile being used:
>>>>>>
>>>>>> farbauti-ce.ofa.iol.unh.edu slots=1
>>>>>> hyperion-ce.ofa.iol.unh.edu slots=1
>>>>>> io-ce.ofa.iol.unh.edu slots=1
>>>>>> jarnsaxa-ce.ofa.iol.unh.edu slots=1
>>>>>> rhea-ce.ofa.iol.unh.edu slots=1
>>>>>> tarqeq-ce.ofa.iol.unh.edu slots=1
>>>>>> tarvos-ce.ofa.iol.unh.edu slots=1
>>>>>>
>>>>>> This seems like a bug and we would like some help to explain and fix
>>>>>> what is happening. The IBTA plugfest saw similar behaviours, so this
>>>>>> should be reproduceable.
>>>>>>
>>>>>> Thanks,
>>>>>> Adam LeBlanc
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users@lists.open-mpi.org
>>>>> https://lists.open-mpi.org/mailman/listinfo/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users@lists.open-mpi.org
>>>> https://lists.open-mpi.org/mailman/listinfo/users
>>> <passing_verbose_output.txt><failing_verbose_output.txt>_______________________________________________
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>>
>> _______________________________________________
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
