>>>>> "Mike" == Mike Dubman <mi...@dev.mellanox.co.il> writes:
Mike> so, it seems you have old ofed w/o this parameter. Can you
Mike> install latest Mellanox ofed? or check which community ofed
Mike> has it?
Rio is using the kernel.org drivers that are part of Ubuntu/3.13.x and
log_num_mtt is not a parameter in those drivers. In fact log_num_mtt
has never been a parameter in the kernel.org sources (just checked the
git commit history). And it's not needed anymore either, since the
following commit (which is also part of OFED 3.12 btw; Mike, seems
Mellanox OFED is behind with this respect):
-----------------------------------------------------------
commit db5a7a65c05867cb6ff5cb6d556a0edfce631d2d
Author: Roland Dreier <rol...@purestorage.com>
List-Post: users@lists.open-mpi.org
Date: Mon Mar 5 10:05:28 2012 -0800
mlx4_core: Scale size of MTT table with system RAM
The current driver defaults to 1M MTT segments, where each segment holds
8 MTT entries. This limits the total memory registered to 8M * PAGE_SIZE
which is 32GB with 4K pages. Since systems that have much more memory
are pretty common now (at least among systems with InfiniBand hardware),
this limit ends up getting hit in practice quite a bit.
Handle this by having the driver allocate at least enough MTT entries to
cover 2 * totalram pages.
Signed-off-by: Roland Dreier <rol...@purestorage.com>
-----------------------------------------------------------
The relevant code segment (drivers/net/ethernet/mellanox/mlx4/profile.c):
-----------------------------------------------------------
/*
* We want to scale the number of MTTs with the size of the
* system memory, since it makes sense to register a lot of
* memory on a system with a lot of memory. As a heuristic,
* make sure we have enough MTTs to cover twice the system
* memory (with PAGE_SIZE entries).
*
* This number has to be a power of two and fit into 32 bits
* due to device limitations, so cap this at 2^31 as well.
* That limits us to 8TB of memory registration per HCA with
* 4KB pages, which is probably OK for the next few months.
*/
si_meminfo(&si);
request->num_mtt =
roundup_pow_of_two(max_t(unsigned, request->num_mtt,
min(1UL << (31 - log_mtts_per_seg),
si.totalram >> (log_mtts_per_seg -
1))));
-----------------------------------------------------------
So the point here is that OpenMPI should check the mlx4 driver versions
and not output false warnings when newer drivers are used. Didn't check
whether this is fixed in the OpenMPI code repositories yet. It's not
fixed in 1.8.2rc4 anyway (static uint64_t calculate_max_reg in
ompi/mca/btl/openib/btl_openib.c). Also, the OpenMPI FAQ should be
corrected accordingly.
Rio as a note for you: You can safely ignore the warning.
Cheers,
Roland
-------
http://www.q-leap.com / http://qlustar.com
--- HPC / Storage / Cloud Linux Cluster OS ---
Mike> On Tue, Aug 19, 2014 at 9:34 AM, Rio Yokota
Mike> <rioyok...@mac.com> wrote:
>> Here is what "modinfo mlx4_core" gives
>>
>> filename:
>>
/lib/modules/3.13.0-34-generic/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
>> version: 2.2-1 license: Dual BSD/GPL description: Mellanox
>> ConnectX HCA low-level driver author: Roland Dreier srcversion:
>> 3AE29A0A6538EBBE9227361 alias:
>> pci:v000015B3d00001010sv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Fsv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Esv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Dsv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Csv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Bsv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000100Asv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001009sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001008sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001007sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001006sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001005sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001004sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001003sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00001002sv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000676Esv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006746sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006764sv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000675Asv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006372sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006750sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006368sv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000673Csv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006732sv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006354sv*sd*bc*sc*i* alias:
>> pci:v000015B3d0000634Asv*sd*bc*sc*i* alias:
>> pci:v000015B3d00006340sv*sd*bc*sc*i* depends: intree: Y vermagic:
>> 3.13.0-34-generic SMP mod_unload modversions signer: Magrathea:
>> Glacier signing key sig_key:
>> 50:0B:C5:C8:7D:4B:11:5C:F3:C1:50:4F:7A:92:E2:33:C6:14:3D:58
>> sig_hashalgo: sha512 parm: debug_level:Enable debug tracing if >
>> 0 (int) parm: msi_x:attempt to use MSI-X if nonzero (int) parm:
>> num_vfs:enable #num_vfs functions if num_vfs > 0
>> num_vfs=port1,port2,port1+2 (array of byte) parm: probe_vf:number
>> of vfs to probe by pf driver (num_vfs > 0)
>> probe_vf=port1,port2,port1+2 (array of byte) parm:
>> log_num_mgm_entry_size:log mgm size, that defines the num of qp
>> per mcg, for example: 10 gives 248.range: 7 <=
>> log_num_mgm_entry_size <= 12. To activate device managed flow
>> steering when available, set to -1 (int) parm:
>> enable_64b_cqe_eqe:Enable 64 byte CQEs/EQEs when the FW supports
>> this (default: True) (bool) parm: log_num_mac:Log2 max number of
>> MACs per ETH port (1-7) (int) parm: log_num_vlan:Log2 max number
>> of VLANs per ETH port (0-7) (int) parm: use_prio:Enable steering
>> by VLAN priority on ETH ports (0/1, default 0) (bool) parm:
>> log_mtts_per_seg:Log2 number of MTT entries per segment (1-7)
>> (int) parm: port_type_array:Array of port types: HW_DEFAULT (0)
>> is default 1 for IB, 2 for Ethernet (array of int) parm:
>> enable_qos:Enable Quality of Service support in the HCA (default:
>> off) (bool) parm: internal_err_reset:Reset device on internal
>> errors if non-zero (default 1, in SRIOV mode default is 0) (int)
>>
>> most likely you installing old ofed which does not have this
>> parameter:
>>
>> try:
>>
>> #modinfo mlx4_core
>>
>> and see if it is there. I would suggest install latest OFED or
>> Mellanox OFED.
>>
>>
>> On Mon, Aug 18, 2014 at 9:53 PM, Rio Yokota <rioyok...@mac.com>
>> wrote:
>>
>>> I get "ofed_info: command not found". Note that I don't install
>>> the entire OFED, but do a component wise installation by doing
>>> "apt-get install infiniband-diags ibutils ibverbs-utils
>>> libmlx4-dev" for the drivers and utilities.
>>>
>>> Hi, what ofed version do you use? (ofed_info -s)
>>>
>>>
>>> On Sun, Aug 17, 2014 at 7:16 PM, Rio Yokota <rioyok...@mac.com>
>>> wrote:
>>>
>>>> I have recently upgraded from Ubuntu 12.04 to 14.04 and OpenMPI
>>>> gives the following warning upon execution, which did not
>>>> appear before the upgrade.
>>>>
>>>> WARNING: It appears that your OpenFabrics subsystem is
>>>> configured to only allow registering part of your physical
>>>> memory. This can cause MPI jobs to run with erratic
>>>> performance, hang, and/or crash.
>>>>
>>>> Everything that I could find on google suggests to change
>>>> log_num_mtt, but I cannot do this for the following reasons:
>>>> 1. There is no log_num_mtt in /sys/module/mlx4_core/parameters/
>>>> 2. Adding "options mlx4_core log_num_mtt=24" to
>>>> /etc/modprobe.d/mlx4.conf doesn't seem to change anything
>>>> 3. I am not sure how I can restart the driver because there is
>>>> no
>>>> "/etc/init.d/openibd" file (I've rebooted the system but it
>>>> didn't do anything to create log_num_mtt)
>>>>
>>>> [Template information]
>>>> 1. OpenFabrics is from the Ubuntu distribution using "apt-get
>>>> install
>>>> infiniband-diags ibutils ibverbs-utils libmlx4-dev"
>>>> 2. OS is Ubuntu 14.04 LTS
>>>> 3. Subnet manager is from the Ubuntu distribution using
>>>> "apt-get install
>>>> opensm"
>>>> 4. Output of ibv_devinfo is:
>>>> hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.10.600
>>>> node_guid: 0002:c903:003d:52b0 sys_image_guid:
>>>> 0002:c903:003d:52b3 vendor_id: 0x02c9 vendor_part_id: 4099
>>>> hw_ver: 0x0 board_id: MT_1100120019 phys_port_cnt: 1 port: 1
>>>> state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5)
>>>> sm_lid: 1 port_lid: 1 port_lmc: 0x00 link_layer: InfiniBand
>>>> 5. Output of ifconfig for IB is
>>>> ib0 Link encap:UNSPEC HWaddr
>>>> 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00 inet
>>>> addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6
>>>> addr: fe80::202:c903:3d:52b1/64 Scope:Link UP BROADCAST RUNNING
>>>> MULTICAST MTU:2044 Metric:1 RX packets:26 errors:0 dropped:0
>>>> overruns:0 frame:0 TX packets:34 errors:0 dropped:16 overruns:0
>>>> carrier:0 collisions:0 txqueuelen:256 RX bytes:5843 (5.8 KB) TX
>>>> bytes:4324 (4.3 KB)
>>>> 6. ulimit -l is "unlimited"
>>>>
>>>> Thanks, Rio _______________________________________________
>>>> users mailing list us...@open-mpi.org Subscription:
