Hi Bill
I have 2.6.X CentOS stock kernel.
I set both parameters.
It works.
Maybe the parameter names may changed in 3.X kernels?
(Which is really bad ...)
You could check if there is more information in:
/sys/module/mlx4_core/parameters/
There seems to be a thread on the list about this (but apparently
no solution):
http://www.open-mpi.org/community/lists/users/2013/02/21430.php
Maybe Mellanox has more information about this?
Gus Correa
On 10/21/2014 08:15 PM, Bill Broadley wrote:
On 10/21/2014 04:18 PM, Gus Correa wrote:
Hi Bill
Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ?
http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
Ah, that helped. Although:
/lib/modules/3.13.0-36-generic/kernel/drivers/net/ethernet/mellanox/mlx4$
modinfo mlx4_core | grep "^parm"
Lists some promising looking parameters:
parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7)
(int)
The FAQ recommends log_num_mtt or num_mtt and NOT log_mtts_per_seg, sadly:
$ modinfo mlx4_core | grep "^parm" | grep mtt
parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7)
(int)
$
Looks like the best I can do is bump log_mtts_per_seg.
I tried:
$ cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core log_num_mtt=24
$
But:
[ 6.691959] mlx4_core: unknown parameter 'log_num_mtt' ignored
I ended up with:
options mlx4_core log_mtts_per_seg=2
I'm hoping that doubles the registerable memory, although I did see a
recommendation to raise it to double the system ram (in this case 64GB ram/128GB
locakable.
Maybe an update to the FAQ is needed?
