-----Original Message----- From: "Thotton, Shijith" <shijith.thot...@cavium.com> Date: Sunday, September 17, 2017 at 11:49 PM To: "Yang, Qiming" <qiming.y...@intel.com>, Harish Patil <harish.pa...@cavium.com> Cc: "Zhang, Helin" <helin.zh...@intel.com>, Gregory Etelson <greg...@weka.io>, "tho...@monjalon.net" <tho...@monjalon.net>, "dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng....@intel.com>, "Thotton, Shijith" <shijith.thot...@cavium.com>, "Hu, Xuekun" <xuekun...@intel.com>, "Yigit, Ferruh" <ferruh.yi...@intel.com>, "Li, Xiaoyun" <xiaoyun...@intel.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file
>On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote: >> Sorry, Gregory and Shijith >> >> I misunderstand your advice. >> If we only remove PCI reset at igbuio_pci_open, not include >>igbuio_pci_release, I think it is a good solution. >> It reserve the functional of this patch and we have test on FVL >>10G/40G/25G, all work fine. >> > >Thanks Qiming. Glad to hear that removing pci_reset_function from >igbuio_pci_open >fixed the issue for Intel NICs. > >Harish, > >Please let us know the behavior with QEDE. Reverting won't be necessary >if the >fix works for all. > >Thanks, >Shijith [Harish] Shijith, Removing pci_reset_function() from open() seems good enough. Thanks. > >> > -----Original Message----- >> > From: Zhang, Helin >> > Sent: Monday, September 18, 2017 10:39 AM >> > To: Yang, Qiming <qiming.y...@intel.com>; Gregory Etelson >> > <greg...@weka.io>; tho...@monjalon.net >> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng....@intel.com>; Thotton, >>Shijith >> > <shijith.thot...@cavium.com>; Hu, Xuekun <xuekun...@intel.com>; Yigit, >> > Ferruh <ferruh.yi...@intel.com>; Patil, Harish >><harish.pa...@cavium.com>; Li, >> > Xiaoyun <xiaoyun...@intel.com> >> > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > Hi guys >> > >> > May I suggest to revert the patch first, as it blocks existing >>functionalities. >> > Then the original authors can add it back with a fix? >> > >> > Regards, >> > Helin >> > >> > -----Original Message----- >> > From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Yang, Qiming >> > Sent: Monday, September 18, 2017 10:21 AM >> > To: Gregory Etelson <greg...@weka.io> >> > Cc: tho...@monjalon.net; dev@dpdk.org; Tan, Jianfeng >> > <jianfeng....@intel.com>; Thotton, Shijith >><shijith.thot...@cavium.com>; Hu, >> > Xuekun <xuekun...@intel.com>; Yigit, Ferruh <ferruh.yi...@intel.com>; >>Patil, >> > Harish <harish.pa...@cavium.com>; Li, Xiaoyun <xiaoyun...@intel.com> >> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > Yes, I have tried, and it works. But I think removing this function >>also makes the >> > patch meaningless. >> > >> > Qiming >> > From: Gregory Etelson [mailto:greg...@weka.io] >> > Sent: Sunday, September 17, 2017 10:50 AM >> > To: Yang, Qiming <qiming.y...@intel.com> >> > Cc: tho...@monjalon.net; dev@dpdk.org; Tan, Jianfeng >> > <jianfeng....@intel.com>; Thotton, Shijith >><shijith.thot...@cavium.com>; Hu, >> > Xuekun <xuekun...@intel.com>; Yigit, Ferruh <ferruh.yi...@intel.com>; >>Patil, >> > Harish <harish.pa...@cavium.com> >> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > >> > Hello Qiming, >> > >> > >> > >> > Did you try to remove PCI reset at igbuio_pci_open ? >> > >> > Reverting the patch entirely will get us back to VF issues >> > >> > >> > >> > Regards, >> > >> > Gregory >> > >> > >> > >> > >> > >> > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: >> > >> > > Hi, Thomas >> > >> > > We meet a urgent issue because of Shijith's patch. >> > >> > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, >>i40e, ixgbe >> > and qede(Harish said) VF all meet initial failure. >> > >> > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 >>igb_uio all >> > works well. >> > >> > > Considering this issue will block our vf driver develop and test >>work, could we >> > revert this patch first and find another appropriate way? >> > >> > > >> > >> > > Qiming >> > >> > > >> > >> > > > -----Original Message----- >> > >> > > > From: Patil, Harish [mailto:harish.pa...@cavium.com] >> > >> > > > Sent: Thursday, September 14, 2017 3:45 AM >> > >> > > > To: Yigit, Ferruh >><ferruh.yi...@intel.com<mailto:ferruh.yi...@intel.com>>; >> > Hu, Xuekun <xuekun...@intel.com<mailto:xuekun...@intel.com>>; >> > >> > > > Thotton, Shijith >> > <shijith.thot...@cavium.com<mailto:shijith.thot...@cavium.com>>; Yang, >> > Qiming >> > >> > > > <qiming.y...@intel.com<mailto:qiming.y...@intel.com>>; Gregory >>Etelson >> > <greg...@weka.io<mailto:greg...@weka.io>> >> > >> > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >> > <jianfeng....@intel.com<mailto:jianfeng....@intel.com>> >> > >> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue >>FLR during open >> > >> > > > and release of device file >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > -----Original Message----- >> > >> > > > From: dev <dev-boun...@dpdk.org<mailto:dev-boun...@dpdk.org>> on >> > behalf of Ferruh Yigit >> > >> > > > <ferruh.yi...@intel.com<mailto:ferruh.yi...@intel.com>> >> > >> > > > Date: Wednesday, September 13, 2017 at 10:06 AM >> > >> > > > To: "Hu, Xuekun" >><xuekun...@intel.com<mailto:xuekun...@intel.com>>, >> > "Thotton, Shijith" >> > >> > > > <shijith.thot...@cavium.com<mailto:shijith.thot...@cavium.com>>, >> > "Yang, Qiming" <qiming.y...@intel.com<mailto:qiming.y...@intel.com>>, >> > >> > > > Gregory Etelson <greg...@weka.io<mailto:greg...@weka.io>> >> > >> > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" >> > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" >> > <jianfeng....@intel.com<mailto:jianfeng....@intel.com>> >> > >> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue >>FLR during open >> > >> > > > and release of device file >> > >> > > > >> > >> > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >> > >> > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel >>driver. >> > >> > > > >> >> > >> > > > >> -----Original Message----- >> > >> > > > >> From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Ferruh >>Yigit >> > >> > > > >> Sent: Wednesday, September 13, 2017 7:04 PM >> > >> > > > >> To: Shijith Thotton >> > >><shijith.thot...@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>>; Yang, >> > >> > > > >>Qiming <qiming.y...@intel.com<mailto:qiming.y...@intel.com>>; >> > Gregory Etelson <greg...@weka.io<mailto:greg...@weka.io>> >> > >> > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >> > <jianfeng....@intel.com<mailto:jianfeng....@intel.com>> >> > >> > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: >>issue FLR >> > >> > > > >>during open and release of device file >> > >> > > > >> >> > >> > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >> > >> > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >> > >> > > > >>>> Hi, Shijith >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> VF init error will happen after apply your patch, error log >>as >> > >> > > > >>>>below. If >> > >> > > > >>>> revert your commit, all things work well. And this issue is >>not >> > >> > > > >>>>only occur >> > >> > > > >>>> in i40 VF but also ixgbe. >> > >> > > > > >> > >> > > > >Hi Qiming, >> > >> > > > > >> > >> > > > >I can reproduce the issue for the case: >> > >> > > > > >> > >> > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >> > >> > > > >2- DPDK app run on host, so both PF and VF are in host. Linux >>driver >> > >> > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host >>and >> > >> > > > >DPDK VF on guest, this works fine) >> > >> > > > > >> > >> > > > >And I confirm the function igbuio_pci_open() added with below >>patch is >> > >> > > > >causing it. >> > >> > > > > >> > >> > > > >igbuio_pci_open() does: >> > >> > > > > pci_reset_function(dev); >> > >> > > > > pci_set_master(dev); >> > >> > > > > >> > >> > > > >Don't know yet root cause of the error, will dig more. >> > >> > > > > >> > >> > > > >>>> >> > >> > > > >>>> Could you help to check it soon? >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >> > >> > > > >>>> >> > >> > > > >>>> EAL: Detected 10 lcore(s) >> > >> > > > >>>> >> > >> > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB >> > >> > > > >>>> >> > >> > > > >>>> EAL: Probing VFIO support... >> > >> > > > >>>> >> > >> > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> >>using >> > >> > > > >>>> unreliable clock cycles ! >> > >> > > > >>>> >> > >> > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >> > >> > > > >>>> >> > >> > > > >>>> EAL: Invalid NUMA socket, default to 0 >> > >> > > > >>>> >> > >> > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf >> > >> > > > >>>> >> > >> > > > >>>> i40evf_init_vf(): init_adminq failed: -53 >> > >> > > > >>>> >> > >> > > > >>>> i40evf_dev_init(): Init vf failed >> > >> > > > >>>> >> > >> > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >> > >> > > > >>>> Author: Shijith Thotton >> > >><shijith.thot...@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>> >> > >> > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 >> > >> > > > >>>> >> > >> > > > >>>> igb_uio: issue FLR during open and release of device file >> > >> > > > >>>> >> > >> > > > >>>> Set UIO info device file operations open and release. Call >>pci >> > >> > > > >>>>reset >> > >> > > > >>>> function inside open and release to clear device state at >>start >> > >> > > > >>>>and end. >> > >> > > > >>>> Copied this behaviour from vfio_pci kernel module code. With >> > >> > > > >>>>this patch, >> > >> > > > >>>> it is not mandatory to issue FLR by PMD's during init and >>close. >> > >> > > > >>>> >> > >> > > > >>>> Bus master enable and disable are added in open and release >> > >> > > > >>>>respectively >> > >> > > > >>>> to take care of device DMA. >> > >> > > > >>>> >> > >> > > > >>>> Signed-off-by: Shijith Thotton >> > >><shijith.thot...@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>> >> > >> > > > >>>> Reviewed-by: Jianfeng Tan >> > <jianfeng....@intel.com<mailto:jianfeng....@intel.com>> >> > >> > > > >>>> Acked-by: Ferruh Yigit >> > <ferruh.yi...@intel.com<mailto:ferruh.yi...@intel.com>> >> > >> > > > >>>> Acked-by: Gregory Etelson >> > <greg...@weka.io<mailto:greg...@weka.io>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> Best Regard, >> > >> > > > >>>> >> > >> > > > >>>> Yang Qiming >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>> >> > >> > > > >>> Hi Yang Qiming, >> > >> > > > >>> >> > >> > > > >>> We didn't face this issue while preparing patch. I think, >>Gregory >> > >> > > > >>> tested it for Intel NICs. Adding more people to the loop for >>help. >> > >> > > > >>> >> > >> > > > >> >> > >> > > > >> I will try to reproduce the issue. >> > >> > > > >> >> > >> > > > >>> Shijith >> > >> > > > >> > >> > > > [Harish] We are seeing similar issue with the bringup of qede VF >>devices due >> > to >> > >> > > > this patch. >> > >> > > > Could we revert this and leave it to individual PMDs to manage it? >> > >> > > > >> >> > >> > > >> > >> > > >> > >>
