Hi Aloÿs Thanks for fixing this issue so fast. ☺ ----------------------------- Best Regards, Sun, Chenmin
From: vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] On Behalf Of Aloys Augustin (aloaugus) via Lists.Fd.Io Sent: Tuesday, September 17, 2019 4:16 PM To: Sun, Chenmin <chenmin....@intel.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Issue with DPDK 19.08 / i40e driver Hi Chenmin, Thanks a lot for digging into this. I found an old issue, VPP-133: https://jira.fd.io/browse/VPP-133, that looks very similar to this bug. Increasing the stack size allocated to CLI processes resolves the issue, so it looks like the inlining increased DPDK’s stack usage. I have pushed a patch to gerrit that fixes this bug: https://gerrit.fd.io/r/c/vpp/+/22074 . Best, Aloÿs On 16 Sep 2019, at 08:12, Sun, Chenmin <chenmin....@intel.com<mailto:chenmin....@intel.com>> wrote: Hi, I've bisected and found this issue is imported by DPDK commit: commit 1f4d55be438b428bed74f2e3dc49cfd6efc3e6fd Author: Maxime Coquelin <maxime.coque...@redhat.com<mailto:maxime.coque...@redhat.com>> Date: Wed May 29 15:04:20 2019 +0200 eal/x86: force inlining of all memcpy and mov helpers Some helpers in the header file are forced inlined other are only inlined, this patch forces inline for all. It will avoid it to be embedded as functions when called multiple times in the same object file. For example, when we added packed ring support in vhost-user library, rte_memcpy_generic got no more inlined. Signed-off-by: Maxime Coquelin <maxime.coque...@redhat.com<mailto:maxime.coque...@redhat.com>> Acked-by: Bruce Richardson <bruce.richard...@intel.com<mailto:bruce.richard...@intel.com>> The segment fault disappeared after I reverted this commit. I also tried with testpmd, which is a DPDK app integrated in DPDK code, and didn't see this issue. That DPDK commit just changed the function declaration from inlie to __rte_always_inline, all seems ok but why brings such a strange issue to VPP(debug mode only). ----------------------------- Best Regards, Sun, Chenmin From: Yang, Zhiyong Sent: Monday, September 16, 2019 2:08 PM To: Sun, Chenmin <chenmin....@intel.com<mailto:chenmin....@intel.com>> Subject: FW: [vpp-dev] Issue with DPDK 19.08 / i40e driver From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion via Lists.Fd.Io Sent: Thursday, September 12, 2019 9:00 PM To: Aloys Augustin (aloaugus) <aloau...@cisco.com<mailto:aloau...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Issue with DPDK 19.08 / i40e driver Florin hit the same issue few weeks ago, don't use debug version of DPDK and it will work OK. it is unlikely vpp issue, as it happens only when dpdk is compiled with -g0. It looks like stack corruption in dpdk. If somebody have cycles, probably bisecting the dpdk between last two releases will give some clue what is wrong. On 12 Sep 2019, at 13:55, Aloys Augustin (aloaugus) via Lists.Fd.Io <aloaugus=cisco....@lists.fd.io<mailto:aloaugus=cisco....@lists.fd.io>> wrote: Hi Damjan, I encountered the same issue on a different host with the same hardware, here is a stack trace: DBGvpp# sh int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count FortyGigabitEthernet5e/0/0 1 down 9000/0/0/0 local0 0 down 0/0/0/0 DBGvpp# set int state FortyGigabitEthernet5e/0/0 up Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault. 0x00007fffb07b9f07 in i40e_asq_send_command (hw=0x7fc1fff9d340, desc=0x7fffb9880cc0, buff=0x7fc2002c3b80, buff_size=16, cmd_details=0x0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_adminq.c:933 933 wr32(hw, hw->aq.asq.tail, hw->aq.asq.next_to_use); (gdb) bt #0 0x00007fffb07b9f07 in i40e_asq_send_command (hw=0x7fc1fff9d340, desc=0x7fffb9880cc0, buff=0x7fc2002c3b80, buff_size=16, cmd_details=0x0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_adminq.c:933 #1 0x00007fffb0812e1c in i40e_aq_remove_macvlan (hw=0x7fc1fff9d340, seid=390, mv_list=0x7fc2002c3b80, count=1, cmd_details=0x0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_common.c:3121 #2 0x00007fffb0992a10 in i40e_remove_macvlan_filters (vsi=0x7fc1fff1aac0, filter=0x7fc1fff1c280, total=1) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:6881 #3 0x00007fffb09bd4b8 in i40e_vsi_delete_mac (vsi=0x7fc1fff1aac0, addr=0x7fffb98856c0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:7319 #4 0x00007fffb0a2172b in i40e_set_default_mac_addr (dev=0x7fffb3069d00 <rte_eth_devices>, mac_addr=0x7fc1fff1a500) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:11962 #5 0x00007fffaffb3491 in rte_eth_dev_mac_restore (dev=0x7fffb3069d00 <rte_eth_devices>, dev_info=0x7fffb9885770) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1357 #6 0x00007fffaffb35d5 in rte_eth_dev_config_restore (dev=0x7fffb3069d00 <rte_eth_devices>, dev_info=0x7fffb9885770, port_id=0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1388 #7 0x00007fffaffb37e9 in rte_eth_dev_start (port_id=0) at /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1435 #8 0x00007fffb2732604 in dpdk_device_start (xd=0x7fffb9842740) at /home/aloaugus/vpp/src/plugins/dpdk/device/common.c:168 #9 0x00007fffb274b25d in dpdk_interface_admin_up_down (vnm=0x7ffff7b64520 <vnet_main>, hw_if_index=1, flags=1) at /home/aloaugus/vpp/src/plugins/dpdk/device/device.c:483 #10 0x00007ffff6c71bb4 in vnet_sw_interface_set_flags_helper (vnm=0x7ffff7b64520 <vnet_main>, sw_if_index=1, flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=(unknown: 0)) at /home/aloaugus/vpp/src/vnet/interface.c:455 #11 0x00007ffff6c71cde in vnet_sw_interface_set_flags (vnm=0x7ffff7b64520 <vnet_main>, sw_if_index=1, flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP) at /home/aloaugus/vpp/src/vnet/interface.c:504 #12 0x00007ffff6c8d3b1 in set_state (vm=0x7ffff66b5dc0 <vlib_global_main>, input=0x7fffb9885f00, cmd=0x7fffb9534e40) at /home/aloaugus/vpp/src/vnet/interface_cli.c:902 #13 0x00007ffff63c7085 in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, input=0x7fffb9885f00, parent_command_index=66) at /home/aloaugus/vpp/src/vlib/cli.c:645 #14 0x00007ffff63c6f1a in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, input=0x7fffb9885f00, parent_command_index=36) at /home/aloaugus/vpp/src/vlib/cli.c:606 #15 0x00007ffff63c6f1a in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, input=0x7fffb9885f00, parent_command_index=0) at /home/aloaugus/vpp/src/vlib/cli.c:606 #16 0x00007ffff63c74b0 in vlib_cli_input (vm=0x7ffff66b5dc0 <vlib_global_main>, input=0x7fffb9885f00, function=0x7ffff646d6de <unix_vlib_cli_output>, function_arg=0) at /home/aloaugus/vpp/src/vlib/cli.c:746 #17 0x00007ffff6473514 in unix_cli_process_input (cm=0x7ffff66b67a0 <unix_cli_main>, cli_file_index=0) at /home/aloaugus/vpp/src/vlib/unix/cli.c:2525 #18 0x00007ffff6474086 in unix_cli_process (vm=0x7ffff66b5dc0 <vlib_global_main>, rt=0x7fffb9875000, f=0x0) at /home/aloaugus/vpp/src/vlib/unix/cli.c:2641 #19 0x00007ffff64148e4 in vlib_process_bootstrap (_a=140736284985712) at /home/aloaugus/vpp/src/vlib/main.c:1468 #20 0x00007ffff5eb7a48 in clib_calljmp () from /home/aloaugus/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.20.01 #21 0x00007fffb8460940 in ?? () #22 0x00007ffff64149ec in vlib_process_startup (vm=0x7ffff66b5dc0 <vlib_global_main>, p=0x259, f=0x0) at /home/aloaugus/vpp/src/vlib/main.c:1490 Backtrace stopped: previous frame inner to this frame (corrupt stack?) Reverting commit b6103105f99e0c7f210a9596f205a1efd21b626f (update of dpdk to 19.08) fixes the issue. Thanks, Aloÿs On 11 Sep 2019, at 21:29, Damjan Marion via Lists.Fd.Io <dmarion=me....@lists.fd.io<mailto:dmarion=me....@lists.fd.io>> wrote: On 11 Sep 2019, at 16:43, Mathias Raoul <mathias.ra...@gmail.com<mailto:mathias.ra...@gmail.com>> wrote: Hello, I have an issue with VPP and i40e driver, when I try to switch the interface to up, the program stop with a segmentation fault. My configuration details are below. It might be a compatibility issue, because the DPDK documentation recommend using the firmware v7 for i40E with DPDK v19.08. But the firmware is not yet available for the Cisco XL710 card. vpp stop in this file : dpdk-19.08/drivers/net/i40/base/i40e_adminq.c:933 When I change dpdk version to 19.05 the bug disappear. DBGvpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count FortyGigabitEthernet5e/0/0 1 down 9000/0/0/0 local0 0 down 0/0/0/0 DBGvpp# set interface state FortyGigabitEthernet5e/0/0 up vl_msg_api_trace_save:252: Message table length 44998 Configuration: VPP : last commit on master : 1146ff4bcd336d8efc19405f1d83914e6115a01f show version verbose Version: v20.01-rc0~171-g1146ff4bc Compiled by: root Compile host: 524b94e75c4d Compile date: Wed Sep 11 12:42:53 UTC 2019 Compile location: /home/mraoul/dev/vpp Compiler: GCC 7.4.0 Current PID: 19052 OS : Ubuntu 18.04.2 LTS Network card : Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)\ There is no enough information in your email to make any conclusion. Can you try to run VPP under gdb and grab traceback, preferably with debug image? -- Damjan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13964): https://lists.fd.io/g/vpp-dev/message/13964 Mute This Topic: https://lists.fd.io/mt/34104216/1643303 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [aloau...@cisco.com<mailto:aloau...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13971): https://lists.fd.io/g/vpp-dev/message/13971 Mute This Topic: https://lists.fd.io/mt/34104216/675642 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com<mailto:dmar...@me.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14000): https://lists.fd.io/g/vpp-dev/message/14000 Mute This Topic: https://lists.fd.io/mt/34104216/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
