On 9/21/23 11:32, Levente Csikor wrote: > Thanks Ilya for the response. > > It turned out that the latest DPDK I can manually compile on the > Bluefield-2 without errors is v20.08. > > After looking at the table you pointed me, my best option was > OvS 2.14.9 with DPDK-v19.11.13. > > Compilations went well without error, now I have the same problem when > attaching the ports to OvS. In fact, even attaching the physical port > fails. > > The way how I initiate OvS is using my custom tailored script > https://github.com/cslev/nvidia-bluefield-ovs-scripts/blob/main/start_ovs.sh > From line 223. > > > After adding the physical port, ovs-vsctl show provides this: > ``` > Bridge ovs_dpdk_br0 > datapath_type: netdev > Port dpdk0 > Interface dpdk0 > type: dpdk > options: {dpdk-devargs="0000:03:00.0"} > error: "Error attaching device '0000:03:00.0' to DPDK" > Port ovs_dpdk_br0 > Interface ovs_dpdk_br0 > type: internal > ``` > > vswitchd log: > ``` > ... > 2023-09-21T09:29:48.626Z|00067|bridge|INFO|bridge ovs_dpdk_br0: added > interface ovs_dpdk_br0 on port 65534 > 2023-09-21T09:29:48.627Z|00068|bridge|INFO|bridge ovs_dpdk_br0: using > datapath ID 0000d62195e14c4e > 2023-09-21T09:29:48.627Z|00069|connmgr|INFO|ovs_dpdk_br0: added service > controller "punix:/usr/local/var/run/openvswitch/ovs_dpdk_br0.mgmt" > 2023-09-21T09:29:48.718Z|00070|dpdk|ERR|EAL: Driver cannot attach the > device (0000:03:00.0) > 2023-09-21T09:29:48.718Z|00071|dpdk|ERR|EAL: Failed to attach device on > primary process > 2023-09-21T09:29:48.719Z|00072|netdev_dpdk|WARN|Error attaching device > '0000:03:00.0' to DPDK > 2023-09-21T09:29:48.719Z|00073|netdev|WARN|dpdk0: could not set > configuration (Invalid argument) > 2023-09-21T09:29:48.719Z|00074|dpdk|ERR|Invalid port_id=32 > 2023-09-21T09:29:57.230Z|00075|memory|INFO|27944 kB peak resident set > size after 10.3 seconds > 2023-09-21T09:29:57.230Z|00076|memory|INFO|handlers:1 ports:1 > revalidators:1 rules:5 > ``` > > > Is there any other log files I can look into for more informational > debug messages?
You may enable additional DPDK debug logs by setting --log-level in the dpdk-extra config. However, your DPDK version is older than a hardware you're using. Meaning, the mlx driver in DPDK 19.11 likely just doesn't recognize the hardware and will not be able to use it. You need to figure out how to build newer versions of DPDK. Also, representor syntax changed a few times in the past, so your script may not work with older versions of DPDK. If you can't build DPDK on the board itself, cross-compiling may be an option: https://doc.dpdk.org/guides/platform/bluefield.html Also OVS 2.14 is likely a poor choice for hardware offloading. I would not recommend anything below 2.17. 3.1 would be a better. > > > Cheers, > levi > > > On Wed, 2023-09-20 at 17:20 +0200, Ilya Maximets wrote: >> On 9/20/23 06:39, Levente Csikor via discuss wrote: >>> Hi All, >>> >>> I have a long lasting problem I have been trying to resolve for >>> quite >>> some time. I am playing around with an NVidia SmartNIC (Bluefield- >>> 2), >>> which has OvS installed by default. It works well with the kernel >>> driver, and even TC hardware offloading is working. >>> >>> I want to experiment with DPDK, though. >>> DPDK is also installed by default on the Bluefield-2. >>> The details of the versions are as follows: >>> >>> OvS 2.17.7 >>> DPDK 22.11.1.4.2 >>> >>> Following the "NVidia tutorials", I manage to add the physical port >>> as >>> a netdev device to an OVS-DPDK bridge, however, adding the virtual >>> function fails. >>> >>> More details about the commands and problem are here: >>> https://forums.developer.nvidia.com/t/error-with-configuring-ovs-dpdk-on-bluefiled-2/256030/4 >>> >>> >>> Anyway, as a last resort, I thought I give a try to install OvS and >>> DPDK from scratch following >>> https://docs.openvswitch.org/en/latest/intro/install/dpdk/ >>> >>> I used the same version for OvS and DPDK; the latter was anyway the >>> one >>> recommended by the OvS documentation. >>> >>> During the `make` process, I encounter several errors I cannot >>> really >>> resolve >>> ``` >>> In file included from lib/dp-packet.h:29, >>> from lib/bfd.c:28: >>> lib/netdev-dpdk.h:95:12: warning: ‘struct rte_flow_tunnel’ declared >>> inside parameter list will not be visible outside of this >>> definition or >>> declaration >>> 95 | struct rte_flow_tunnel *tunnel OVS_UNUSED, >>> | ^~~~~~~~~~~~~~~ >>> lib/netdev-dpdk.h:106:42: warning: ‘struct rte_flow_tunnel’ >>> declared >>> inside parameter list will not be visible outside of this >>> definition or >>> declaration >>> 106 | struct rte_flow_tunnel >>> *tunnel OVS_UNUSED, >>> | ^~~~~~~~~~~~~~~ >>> lib/netdev-dpdk.h:119:12: warning: ‘struct rte_flow_restore_info’ >>> declared inside parameter list will not be visible outside of this >>> definition or declaration >>> 119 | struct rte_flow_restore_info *info OVS_UNUSED, >>> | ^~~~~~~~~~~~~~~~~~~~~ >>> In file included from lib/bfd.c:28: >>> lib/dp-packet.h:61:40: error: ‘RTE_MBUF_F_RX_RSS_HASH’ undeclared >>> here >>> (not in a function) >>> 61 | DEF_OL_FLAG(DP_PACKET_OL_RSS_HASH, >>> RTE_MBUF_F_RX_RSS_HASH, >>> 0x1), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:63:41: error: ‘RTE_MBUF_F_RX_FDIR_ID’ undeclared >>> here >>> (not in a function) >>> 63 | DEF_OL_FLAG(DP_PACKET_OL_FLOW_MARK, >>> RTE_MBUF_F_RX_FDIR_ID, >>> 0x2), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:65:47: error: ‘RTE_MBUF_F_RX_L4_CKSUM_BAD’ >>> undeclared >>> here (not in a function) >>> 65 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_BAD, >>> RTE_MBUF_F_RX_L4_CKSUM_BAD, 0x4), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:67:47: error: ‘RTE_MBUF_F_RX_IP_CKSUM_BAD’ >>> undeclared >>> here (not in a function) >>> 67 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_BAD, >>> RTE_MBUF_F_RX_IP_CKSUM_BAD, 0x8), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:69:48: error: ‘RTE_MBUF_F_RX_L4_CKSUM_GOOD’ >>> undeclared >>> here (not in a function) >>> 69 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_GOOD, >>> RTE_MBUF_F_RX_L4_CKSUM_GOOD, >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:72:48: error: ‘RTE_MBUF_F_RX_IP_CKSUM_GOOD’ >>> undeclared >>> here (not in a function) >>> 72 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_GOOD, >>> RTE_MBUF_F_RX_IP_CKSUM_GOOD, >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:75:42: error: ‘RTE_MBUF_F_TX_TCP_SEG’ undeclared >>> here >>> (not in a function); did you mean ‘RTE_MBUF_MAX_NB_SEGS’? >>> 75 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_SEG, >>> RTE_MBUF_F_TX_TCP_SEG, >>> 0x40), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:77:39: error: ‘RTE_MBUF_F_TX_IPV4’ undeclared here >>> (not >>> in a function) >>> 77 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV4, RTE_MBUF_F_TX_IPV4, >>> 0x80), >>> | ^~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:79:39: error: ‘RTE_MBUF_F_TX_IPV6’ undeclared here >>> (not >>> in a function) >>> 79 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV6, RTE_MBUF_F_TX_IPV6, >>> 0x100), >>> | ^~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:81:44: error: ‘RTE_MBUF_F_TX_TCP_CKSUM’ undeclared >>> here >>> (not in a function) >>> 81 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_CKSUM, >>> RTE_MBUF_F_TX_TCP_CKSUM, 0x200), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:83:44: error: ‘RTE_MBUF_F_TX_UDP_CKSUM’ undeclared >>> here >>> (not in a function) >>> 83 | DEF_OL_FLAG(DP_PACKET_OL_TX_UDP_CKSUM, >>> RTE_MBUF_F_TX_UDP_CKSUM, 0x400), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> lib/dp-packet.h:85:45: error: ‘RTE_MBUF_F_TX_SCTP_CKSUM’ undeclared >>> here (not in a function) >>> 85 | DEF_OL_FLAG(DP_PACKET_OL_TX_SCTP_CKSUM, >>> RTE_MBUF_F_TX_SCTP_CKSUM, 0x800), >>> | >>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>> lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >>> 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = >>> DPDK_DEF >>> | >>> ^~~~~~~~ >>> make[2]: *** [Makefile:4645: lib/bfd.lo] Error 1 >>> make[2]: Leaving directory '/opt/ovs-2.17.7' >>> make[1]: *** [Makefile:5409: all-recursive] Error 1 >>> make[1]: Leaving directory '/opt/ovs-2.17.7' >>> make: *** [Makefile:3079: all] Error 2 >>> ``` >>> >>> Asking google did not help at all :( >>> Did anyone encounter such errors? >> >> Nvidia is using their closed-source versions of OVS and DPDK, so the >> versions you provided are not what you need. >> >> Upstream OVS 2.17 works only with DPDK 21.11. See the "What DPDK >> version >> does each Open vSwitch release work with?" section of releases FAQ: >> https://docs.openvswitch.org/en/latest/faq/releases/ >> >> DPDK 22.11 is supported starting with OVS 3.1. But I would guess >> that >> Nvidia's DPDK 22.11 is not actually an upstream DPDK 22.11. >> >> Best regards, Ilya Maximets. >> >>> >>> btw. the Ethernet ports/hw that the Bluefield-2 has is a >>> Mellanox/Nvidia ConnectX-6, using the mlx5_core driver. >>> >>> >>> All comments are welcome :) >>> >>> Thanks, >>> Levi > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
