Hi Chenmin, Thanks a lot for digging into this. I found an old issue, VPP-133: https://jira.fd.io/browse/VPP-133 <https://jira.fd.io/browse/VPP-133>, that looks very similar to this bug. Increasing the stack size allocated to CLI processes resolves the issue, so it looks like the inlining increased DPDK’s stack usage. I have pushed a patch to gerrit that fixes this bug: https://gerrit.fd.io/r/c/vpp/+/22074 <https://gerrit.fd.io/r/c/vpp/+/22074> .
Best, Aloÿs > On 16 Sep 2019, at 08:12, Sun, Chenmin <chenmin....@intel.com> wrote: > > Hi, <> > > I've bisected and found this issue is imported by DPDK commit: > > commit 1f4d55be438b428bed74f2e3dc49cfd6efc3e6fd > Author: Maxime Coquelin <maxime.coque...@redhat.com > <mailto:maxime.coque...@redhat.com>> > Date: Wed May 29 15:04:20 2019 +0200 > > eal/x86: force inlining of all memcpy and mov helpers > > Some helpers in the header file are forced inlined other are > only inlined, this patch forces inline for all. > > It will avoid it to be embedded as functions when called multiple > times in the same object file. For example, when we added packed > ring support in vhost-user library, rte_memcpy_generic got no > more inlined. > > Signed-off-by: Maxime Coquelin <maxime.coque...@redhat.com > <mailto:maxime.coque...@redhat.com>> > Acked-by: Bruce Richardson <bruce.richard...@intel.com > <mailto:bruce.richard...@intel.com>> > > The segment fault disappeared after I reverted this commit. > I also tried with testpmd, which is a DPDK app integrated in DPDK code, and > didn't see this issue. > > That DPDK commit just changed the function declaration from inlie to > __rte_always_inline, all seems ok but why brings such a strange issue to > VPP(debug mode only). > > ----------------------------- > Best Regards, > Sun, Chenmin > > From: Yang, Zhiyong > Sent: Monday, September 16, 2019 2:08 PM > To: Sun, Chenmin <chenmin....@intel.com> > Subject: FW: [vpp-dev] Issue with DPDK 19.08 / i40e driver > > > > <>From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion > via Lists.Fd.Io > Sent: Thursday, September 12, 2019 9:00 PM > To: Aloys Augustin (aloaugus) <aloau...@cisco.com <mailto:aloau...@cisco.com>> > Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > Subject: Re: [vpp-dev] Issue with DPDK 19.08 / i40e driver > > > Florin hit the same issue few weeks ago, don't use debug version of DPDK and > it will work OK. > it is unlikely vpp issue, as it happens only when dpdk is compiled with -g0. > It looks like stack corruption in dpdk. If somebody have cycles, probably > bisecting the dpdk > between last two releases will give some clue what is wrong. > > > On 12 Sep 2019, at 13:55, Aloys Augustin (aloaugus) via Lists.Fd.Io > <aloaugus=cisco....@lists.fd.io <mailto:aloaugus=cisco....@lists.fd.io>> > wrote: > > Hi Damjan, > > I encountered the same issue on a different host with the same hardware, here > is a stack trace: > > DBGvpp# sh int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > FortyGigabitEthernet5e/0/0 1 down 9000/0/0/0 > local0 0 down 0/0/0/0 > DBGvpp# set int state FortyGigabitEthernet5e/0/0 up > > Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault. > 0x00007fffb07b9f07 in i40e_asq_send_command (hw=0x7fc1fff9d340, > desc=0x7fffb9880cc0, buff=0x7fc2002c3b80, buff_size=16, cmd_details=0x0) > at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_adminq.c:933 > 933 wr32(hw, hw->aq.asq.tail, > hw->aq.asq.next_to_use); > (gdb) bt > #0 0x00007fffb07b9f07 in i40e_asq_send_command (hw=0x7fc1fff9d340, > desc=0x7fffb9880cc0, buff=0x7fc2002c3b80, buff_size=16, cmd_details=0x0) > at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_adminq.c:933 > #1 0x00007fffb0812e1c in i40e_aq_remove_macvlan (hw=0x7fc1fff9d340, > seid=390, mv_list=0x7fc2002c3b80, count=1, cmd_details=0x0) > at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/base/i40e_common.c:3121 > #2 0x00007fffb0992a10 in i40e_remove_macvlan_filters (vsi=0x7fc1fff1aac0, > filter=0x7fc1fff1c280, total=1) at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:6881 > #3 0x00007fffb09bd4b8 in i40e_vsi_delete_mac (vsi=0x7fc1fff1aac0, > addr=0x7fffb98856c0) at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:7319 > #4 0x00007fffb0a2172b in i40e_set_default_mac_addr (dev=0x7fffb3069d00 > <rte_eth_devices>, mac_addr=0x7fc1fff1a500) at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/drivers/net/i40e/i40e_ethdev.c:11962 > #5 0x00007fffaffb3491 in rte_eth_dev_mac_restore (dev=0x7fffb3069d00 > <rte_eth_devices>, dev_info=0x7fffb9885770) at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1357 > #6 0x00007fffaffb35d5 in rte_eth_dev_config_restore (dev=0x7fffb3069d00 > <rte_eth_devices>, dev_info=0x7fffb9885770, port_id=0) > at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1388 > #7 0x00007fffaffb37e9 in rte_eth_dev_start (port_id=0) at > /home/aloaugus/vpp/build-root/build-vpp_debug-native/external/dpdk-19.08/lib/librte_ethdev/rte_ethdev.c:1435 > #8 0x00007fffb2732604 in dpdk_device_start (xd=0x7fffb9842740) at > /home/aloaugus/vpp/src/plugins/dpdk/device/common.c:168 > #9 0x00007fffb274b25d in dpdk_interface_admin_up_down (vnm=0x7ffff7b64520 > <vnet_main>, hw_if_index=1, flags=1) at > /home/aloaugus/vpp/src/plugins/dpdk/device/device.c:483 > #10 0x00007ffff6c71bb4 in vnet_sw_interface_set_flags_helper > (vnm=0x7ffff7b64520 <vnet_main>, sw_if_index=1, > flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=(unknown: 0)) at > /home/aloaugus/vpp/src/vnet/interface.c:455 > #11 0x00007ffff6c71cde in vnet_sw_interface_set_flags (vnm=0x7ffff7b64520 > <vnet_main>, sw_if_index=1, flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP) at > /home/aloaugus/vpp/src/vnet/interface.c:504 > #12 0x00007ffff6c8d3b1 in set_state (vm=0x7ffff66b5dc0 <vlib_global_main>, > input=0x7fffb9885f00, cmd=0x7fffb9534e40) at > /home/aloaugus/vpp/src/vnet/interface_cli.c:902 > #13 0x00007ffff63c7085 in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 > <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, > input=0x7fffb9885f00, parent_command_index=66) at > /home/aloaugus/vpp/src/vlib/cli.c:645 > #14 0x00007ffff63c6f1a in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 > <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, > input=0x7fffb9885f00, parent_command_index=36) at > /home/aloaugus/vpp/src/vlib/cli.c:606 > #15 0x00007ffff63c6f1a in vlib_cli_dispatch_sub_commands (vm=0x7ffff66b5dc0 > <vlib_global_main>, cm=0x7ffff66b5fd0 <vlib_global_main+528>, > input=0x7fffb9885f00, parent_command_index=0) at > /home/aloaugus/vpp/src/vlib/cli.c:606 > #16 0x00007ffff63c74b0 in vlib_cli_input (vm=0x7ffff66b5dc0 > <vlib_global_main>, input=0x7fffb9885f00, function=0x7ffff646d6de > <unix_vlib_cli_output>, function_arg=0) at > /home/aloaugus/vpp/src/vlib/cli.c:746 > #17 0x00007ffff6473514 in unix_cli_process_input (cm=0x7ffff66b67a0 > <unix_cli_main>, cli_file_index=0) at > /home/aloaugus/vpp/src/vlib/unix/cli.c:2525 > #18 0x00007ffff6474086 in unix_cli_process (vm=0x7ffff66b5dc0 > <vlib_global_main>, rt=0x7fffb9875000, f=0x0) at > /home/aloaugus/vpp/src/vlib/unix/cli.c:2641 > #19 0x00007ffff64148e4 in vlib_process_bootstrap (_a=140736284985712) at > /home/aloaugus/vpp/src/vlib/main.c:1468 > #20 0x00007ffff5eb7a48 in clib_calljmp () from > /home/aloaugus/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.20.01 > #21 0x00007fffb8460940 in ?? () > #22 0x00007ffff64149ec in vlib_process_startup (vm=0x7ffff66b5dc0 > <vlib_global_main>, p=0x259, f=0x0) at /home/aloaugus/vpp/src/vlib/main.c:1490 > Backtrace stopped: previous frame inner to this frame (corrupt stack?) > > > Reverting commit b6103105f99e0c7f210a9596f205a1efd21b626f (update of dpdk to > 19.08) fixes the issue. > > Thanks, > Aloÿs > > > On 11 Sep 2019, at 21:29, Damjan Marion via Lists.Fd.Io > <dmarion=me....@lists.fd.io <mailto:dmarion=me....@lists.fd.io>> wrote: > > > > > On 11 Sep 2019, at 16:43, Mathias Raoul <mathias.ra...@gmail.com > <mailto:mathias.ra...@gmail.com>> wrote: > > Hello, > > I have an issue with VPP and i40e driver, when I try to switch the interface > to up, the program stop with a segmentation fault. My configuration details > are below. > > It might be a compatibility issue, because the DPDK documentation recommend > using the firmware v7 for i40E with DPDK v19.08. But the firmware is not yet > available for the Cisco XL710 card. > vpp stop in this file : dpdk-19.08/drivers/net/i40/base/i40e_adminq.c:933 > When I change dpdk version to 19.05 the bug disappear. > > DBGvpp# show int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > FortyGigabitEthernet5e/0/0 1 down 9000/0/0/0 > local0 0 down 0/0/0/0 > DBGvpp# set interface state FortyGigabitEthernet5e/0/0 up > vl_msg_api_trace_save:252: Message table length 44998 > > > Configuration: > VPP : last commit on master : 1146ff4bcd336d8efc19405f1d83914e6115a01f > > show version verbose > Version: v20.01-rc0~171-g1146ff4bc > Compiled by: root > Compile host: 524b94e75c4d > Compile date: Wed Sep 11 12:42:53 UTC 2019 > Compile location: /home/mraoul/dev/vpp > Compiler: GCC 7.4.0 > Current PID: 19052 > > OS : Ubuntu 18.04.2 LTS > > Network card : Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ > (rev 02)\ > > There is no enough information in your email to make any conclusion. Can you > try to run VPP under gdb and grab traceback, preferably with debug image? > > -- > Damjan > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13964): https://lists.fd.io/g/vpp-dev/message/13964 > <https://lists.fd.io/g/vpp-dev/message/13964> > Mute This Topic: https://lists.fd.io/mt/34104216/1643303 > <https://lists.fd.io/mt/34104216/1643303> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [aloau...@cisco.com > <mailto:aloau...@cisco.com>] > -=-=-=-=-=-=-=-=-=-=-=- > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13971): https://lists.fd.io/g/vpp-dev/message/13971 > <https://lists.fd.io/g/vpp-dev/message/13971> > Mute This Topic: https://lists.fd.io/mt/34104216/675642 > <https://lists.fd.io/mt/34104216/675642> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [dmar...@me.com > <mailto:dmar...@me.com>] > -=-=-=-=-=-=-=-=-=-=-=-
smime.p7s
Description: S/MIME cryptographic signature
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13999): https://lists.fd.io/g/vpp-dev/message/13999 Mute This Topic: https://lists.fd.io/mt/34104216/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
