This email is to document an insidious (incorrect data, no error or warning)
VFIO bug found when using the Intel IOMMU to perform DMA transfers; and the
associated workaround.
There may be security implications (unsure).
/sys/devices/virtual/iommu/dmar0/intel-iommu/version: 1:0
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap: d2008c40660462
Linux xxxxx.ibm.com 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST
2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.5 (Ootpa)
In our testing of VFIO DMA to an FPGA card in rootless mode, we discovered a
glitch where DMA data are transferred to/from the incorrect page. It
appears timing based. Under some specific conditions the test could trigger
the bug every loop. Sometimes the bug would only emerge after 20+ minutes
of testing.
Basics of test:
Get memory with mmap(anonymous): size can change.
VFIO_IOMMU_MAP_DMA with a block of memory, fixed IOVA.
Fill memory with pattern.
Do DMA transfer to FPGA from memory at IOVA.
Do DMA transfer from FPGA to memory at IOVA+offset.
Compare memory to ensure match. Miscompare is bug.
VFIO_IOMMU_UNMAP_DMA
unmap()
Repeat.
Using the fixed IOVA address* caused sporadic memory miscompares. The
nature of the miscompares is that the received data was mixed with pages
that had been returned by mmap in a *previous* loop.
Workaround: Randomizing the IOVA eliminated the memory miscompares.
Hypothesis/conjecture: Possible race condition in UNMAP_DMA such that pages
can be released/munlocked *after* the MAP_DMA with the same IOVA has
occurred.
Suggestion: Document issue when using fixed IOVA, or fix if security is a
concern.
Daniel F. Smith
dfsm...@us.ibm.com
* We cannot use physical page address for the IOVA since we are running
without root, so /proc/pagemap is blanked out. We also cannot use the VMA
as the IOVA since MAP_DMA only permits us up to bit 39 in the IOVA.
VMA = virtual memory address (process space)
IOVA = IOV / IOMMU address
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
