> -----Original Message----- > From: Burakov, Anatoly <anatoly.bura...@intel.com> > Sent: Tuesday, July 9, 2019 7:21 PM > To: Jerin Jacob Kollanukkaran <jer...@marvell.com>; David Marchand > <david.march...@redhat.com> > Cc: dev <dev@dpdk.org>; Thomas Monjalon <tho...@monjalon.net>; Ben > Walker <benjamin.wal...@intel.com> > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] bus/pci: fix IOVA as VA mode > selection > > On 09-Jul-19 2:30 PM, Burakov, Anatoly wrote: > > On 09-Jul-19 1:11 PM, Jerin Jacob Kollanukkaran wrote: > >>> -----Original Message----- > >>> From: Burakov, Anatoly <anatoly.bura...@intel.com> > >>> Sent: Tuesday, July 9, 2019 5:10 PM > >>> To: Jerin Jacob Kollanukkaran <jer...@marvell.com>; David Marchand > >>> <david.march...@redhat.com> > >>> Cc: dev <dev@dpdk.org>; Thomas Monjalon <tho...@monjalon.net>; > Ben > >>> Walker <benjamin.wal...@intel.com> > >>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH] bus/pci: fix IOVA as VA > >>> mode selection > >>>>>> ________________________________________ > >>>>>> > >>>>>> On Mon, Jul 8, 2019 at 4:25 PM <mailto:jer...@marvell.com> wrote: > >>>>>> From: Jerin Jacob <mailto:jer...@marvell.com> > >>>>>> > >>>>>> Existing logic fails to select IOVA mode as VA if driver request > >>>>>> to enable IOVA as VA. > >>>>>> > >>>>>> IOVA as VA has more strict requirement than other modes, so > >>>>>> enabling positive logic for IOVA as VA selection. > >>>>>> > >>>>>> This patch also updates the default IOVA mode as PA for PCI > >>>>>> devices as it has to deal with DMA engines unlike the virtual > >>>>>> devices that may need only IOVA as DC. > >>>>>> > >>>>>> We have three cases: > >>>>>> - driver/hw supports IOVA as PA only > >>>>>> > >>>>>> [Jerin] It is not driver cap, it is more of system cap(IOMMU vs > >>>>>> non IOMMU). We are already addressing that case > >>>>> > >>>>> I don't get how this works. How does "system capability" affect > >>>>> what the device itself supports? Are we to assume that *all* > >>>>> hardware support IOVA as VA by default? "System capability" is > >>>>> more of a bus issue than an individual device issue, is it not? > >>>> > >>>> What I meant is, supporting VA vs PA is function of IOMMU(not the > >>>> device > >>> attribute). > >>>> Ie. Device makes the bus master request, if IOMMU available and > >>>> enabled in the SYSTEM , It goes over IOMMU and translate the IOVA > >>>> to > >>> physical address. > >>>> > >>>> Another way to put is, Is there any _PCIe_ device which > >>>> need/requires RTE_PCI_DRV_NEED_IOVA_AS_PA in > >>>> rte_pci_driver.drv_flags > >>>> > >>>> > >>> > >>> Previously, as far as i can tell, the flag was used to indicate > >>> support for IOVA as VA mode, not *requirement* for IOVA as VA mode. > >>> For example, there are multiple patches [1][2][3][4] (i'm sure i can > >>> find more!) that added IOVA as VA support to various drivers, and > >>> they all were worded it in this exact way > >>> - "support for IOVA as VA mode", not "require IOVA as VA mode". As > >>> far as i can tell, none of these drivers *require* IOVA as VA mode - > >>> they merely use this flag to indicate support for it. > >> > >> Some class of devices NEED IOVA as VA for performance reasons. > >> Specially the devices has HW mempool allocators. On those devices If > >> we don’t use IOVA as VA, Upon getting packet from device, It needs to > >> go over > >> rte_mem_iova2virt() per > >> packet see driver/net/dppa2. Which has real performance issue. > > > > I wouldn't classify this as "needing" IOVA. "Need" implies it cannot > > work without it, whereas in this case it's more of a "highly > > recommended" rather than "need". > > > >>> > >>> Now suddenly it turns out that someone somewhere "knew" that "IOVA > >>> as VA" flag in PCI drivers is supposed to indicate *requirement* and > >>> not support, and it appears that this knowledge was not communicated > >>> nor documented anywhere, and is now treated as common knowledge. > >> > >> I think, the confusion here is, I was under impression that # If > >> device supports IOVA as VA and system runs with IOMMU then the dpdk > >> should run in IOVA as VA mode. > >> If above statement true then we don’t really need a new flag. > > > > Exactly. And the flag used to indicate that the device *supports* IOVA > > as VA, not that it *requires* it. > > ...unless the driver itself is written in such a way as to simply not support > VA > to PA lookups
Yes. > - in that case, the above suggested way of simply not indicating > IOVA as PA support would fix the issue in that it will require the device to > either work in IOVA as VA mode, or fail to initialize. Current semantics of > only > having one flag do not distinguish between "can do both PA and VA" and > "can only do VA" - hence the suggestion of adding an additional flag > indicating IOVA as PA support. Currently all device can support "can do both PA and VA" but system limits through vfio-nommu or Igb_uio or KNI it fallback to PA So question comes what we do with new flag in pci_device_iova_mode() In my view: pci_device_iova_mode() can return RTE_IOVA_PA as default for PCI device. if PCIe device supports IOVA_AS_VA, pci_device_iova_mode() needs to return RTE_IOVA_VA if "SYSTEM" supports it In this context, What will be the responsibility of new flag? Or How do you want to change the behavior of pci_device_iova_mode() > > -- > Thanks, > Anatoly
