Hi Adrien, I've copied dev at dpdk.
I re-installed Mellanox OFED, and then re-compiled DPDK without the flags for the static library. I'm running Ubuntu 14.04. However, I still get the same error message (below). I then tried re-installing the Mellanox libraries on dpdk.org, and compiled another copy of DPDK with them statically linked. This appears to work perfectly. I no longer get the permission issues I was getting before, possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+dpdk.org libs statically linked have the same issue: EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1003 librte_pmd_mlx4 libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0 PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? EAL: No probed ethernet devices The OFED install was successful (and it apparently removes/overwrites old versions of the libraries). The installer also checks the firmware versions, and it says I'm on the latest. Loading the kernel modules is also successful, so I can't for the life of me figure out what this issue is now. Again, just to be clear, the full list of steps was: 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel modules 5. Run testpmd Best, Raghav On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <adrien.mazarguil at 6wind.com> wrote: > Hi Raghav, > > On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: > > Hi Adrien, > > > > Thanks for your reply - I hope it's OK that I'm emailing you off-list. > I'm > > a grad student at Princeton working on a research project. > > It's fine, but let's discuss on the ML as much as possible next time so the > community can benefit from it. Since very few people use this driver at the > moment (DPDK 2.0 is not even out yet), I have my own questions for you. > > > Given your patch (which was very helpful, thanks), here is the series of > > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G > > > > 1. Compile and install the mellanox-specific libraries listed here: > > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ > > This is fine, but did you use the provided Makefile to compile these > libraries statically? If so, no issues? > > Also you did not install Mellanox OFED? You should, because updated kernel > modules come from that distribution. It won't work properly (or at all) > without them. > > The libraries from DPDK.org are actually optional once you've installed > Mellanox OFED. They only improve performance. Remember to update firmwares > also. > > > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS > > 3. Create a new configuration which includes > > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag. > > 4. Compile DPDK 2.0 with the new configuration using setup.sh > > 5. Set up hugepages using setup.sh > > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib > > 7. Run testpmd with the right PCI bus address > > > > However, I consistently get: > > sudo: unable to execute ./testpmd: Permission denied > > Now that doesn't make sense, isn't that file missing permissions or > something? Are you running RHEL with a restrictive SELinux policy? > Is sudo configured to allow you to run testpmd? > > In any case, I don't think it's related to librte_pmd_mlx4. > > > I tried taking the interface down with ifconfig because > > (./tools/dpdk_nic_bind.py --status shows the interface as active), but > that > > didn't change anything. > > No, you should leave the interfaces up, it's not supposed to disappear, the > PMD will bring them back up anyway. Actually the Linux kernel interface > remains fully functional even while DPDK applications are running. You can > send packets out of it but as long as both the kernel and DPDK applications > share the same MAC address, only DPDK applications will receive replies. If > you change the MAC address on the kernel interface however, you'll be able > to use it normally. > > > I tried installing the mellanox libraries globally, but then I see these > > error messages: > > > > ------- > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': > > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, > version > > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': > > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, > > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > > reference > > libibverbs: Warning: no userspace device-specific driver found for > > /sys/class/infiniband_verbs/uverbs0 > > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > > EAL: No probed ethernet devices > > Interactive-mode selected > > EAL: Error - exiting with code: 1 > > Cause: Creation of mbuf pool for socket 0 failed > > -------- > > That's because the libibverbs version you're using is not compatible with > the userspace drivers installed on your system (libibverbs uses dlopen() to > load liblmx4). Even though you've installed both from DPDK.org, seems you > have another version installed by your distribution packages. > > If you intend to use dynamic libraries, you should uninstall libibverbs, > libmlx4 and related packages first. > > Regards, > > -- > Adrien Mazarguil > 6WIND >
