OK, I found a machine in our local collection that couldn't get by with this patch, so I will submitting a follow up that forces rte_eval_iova_mode() into PA mode.
On Mon, Nov 27, 2017 at 7:58 AM, Chas Williams <3ch...@gmail.com> wrote: > > > On Mon, Nov 27, 2017 at 4:30 AM, Zhang, Qi Z <qi.z.zh...@intel.com> wrote: > >> Hi William: >> >> > -----Original Message----- >> > From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Chas Williams >> > Sent: Saturday, November 25, 2017 7:57 AM >> > To: dev@dpdk.org >> > Cc: skh...@vmware.com; Chas Williams <3ch...@gmail.com>; Chas >> > Williams <ch...@att.com> >> > Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual >> address fails >> > >> > Some machines appear to have buggy DMAR mappings. A typical mapping >> > error looks like: >> > >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > DMAR: intel_iommu_map: iommu width (39) is not sufficient for the >> > mapped address (7fc4fa800000) >> > >> I met the same issue on some intel atom platform, the root cause is IOMMU >> only support 39 bit virtual address. >> Not sure retry with physical address will be the right fix. I saw >> rte_eal_iova_mode is called at other place, it still take the virtual >> address as the mapped result, does that break something? >> So far the workaround may works by using --virtbase-addr to assign a >> address in range explicitly (for example 0x70,0000,0000) >> Regards >> Qi >> > > I don't think the IOVA usage elsewhere is an issue since I limited my > changes to reworking what was done in commit e85a919286d2 which appears to > be what broken things for me. > > It's not clear that passing --base-virtaddr is a workable solution. > First, it's only a hint to mmap(). The kernel really can do whatever it > wants. And then, what values do I pick? If one value fails, do I just > restart and keep trying new values until it succeeds? I just want to fall > back to the previous behavior. I know the physical mapping will succeed > since the DMAR tables are limited to 36 bits in width. > > And yes, I should update the commit to blame the IOMMU width. > > >> >> > To work around this, attempt to do a physical address mapping if the >> virtual >> > address mapping fails. >> > >> > Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping") >> > >> > Signed-off-by: Chas Williams <ch...@att.com> >> > --- >> > lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++ >> > 1 file changed, 8 insertions(+) >> > >> > diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > b/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > index 58f0123..6250676 100644 >> > --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c >> > @@ -35,6 +35,7 @@ >> > #include <fcntl.h> >> > #include <unistd.h> >> > #include <sys/ioctl.h> >> > +#include <stdbool.h> >> > >> > #include <rte_log.h> >> > #include <rte_memory.h> >> > @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd) >> > /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ >> > for (i = 0; i < RTE_MAX_MEMSEG; i++) { >> > struct vfio_iommu_type1_dma_map dma_map; >> > + int retried = false; >> > >> > if (ms[i].addr == NULL) >> > break; >> > @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd) >> > dma_map.iova = ms[i].iova; >> > dma_map.flags = VFIO_DMA_MAP_FLAG_READ | >> > VFIO_DMA_MAP_FLAG_WRITE; >> > >> > +retry: >> > ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, >> > &dma_map); >> > >> > if (ret) { >> > + if (!retried && rte_eal_iova_mode() == >> RTE_IOVA_VA) { >> > + dma_map.iova = ms[i].iova; >> >> >> > + retried = true; >> > + goto retry; >> > + } >> > RTE_LOG(ERR, EAL, " cannot set up DMA remapping, >> " >> > "error %i (%s)\n", errno, >> > strerror(errno)); >> > -- >> > 2.9.5 >> Regards >> Qi >> > >
