Also, is there any alternative to /proc/pid/pagemap ? I can’t get the contents for the qemu process, guessing because of the get_user_page pins.
On Mon, Jun 6, 2016 at 11:53 AM Colin Godsey <crgod...@gmail.com> wrote: > I was able to reproduce the issue again by enabling KSM and THP, with > aggressive settings for KSM. This time it appeared on one VM when the other > hibernated itself (has also happened with them both running previously). > The issue manifested itself first as two driver crashes (very uncommon for > me, but ignored for this test). Final issue was the triangle-tearing I was > seeing before. Was about the 1hr mark which is rather fast. There were ~300 > shared pages at the time, so assuming it happened upon unsharing the pages. > > I’ve confirmed that THP + no KSM works fine, as it should in v4.4. The > last test I need to do is KSM with no THP. Considering these are both > remapping modules using slightly different methods, I wouldn’t be surprised > if there was a possibility of some sort of ref-counting issue when used > together (combined with the long-held get_user_page for VFIO). KSM and THPs > ref counting in 4.4 is really hard to follow, and they were both entirely > redone in 4.5. > > Looking over the KSM code, THP, VFIO and various other modules… it does > appear that get_user_pages for long-held pins is rather brittle, and the > THP and KSM code doesnt appear to strengthen any of its guarantees. > > KSM does appear to have some really messy count-checking logic around its > write_protection function, including special post-race checks looking for > O_DIRECT. > > I… think I found all the /proc calls I need to catch any mapping that > might be occurring (smaps, maps). Is there a better way, or any more > details proc calls? Also, all IOMMU DMA regions should show in the map > under vfio correct (for the DMAR)? > > System background- > > ubuntu ‘lowlatency’ kernel 4.4.0-22-lowlatency (voluntary kernel > preempt, irq threads, etc). Confirmed artifacts happen on the generic > kernel also. > skylake with updated microcode (hopefully fixing the p-state and hang > issues) > Single dye, single NUMA region. > > On Tue, May 31, 2016 at 4:54 PM Colin Godsey <crgod...@gmail.com> wrote: > >> Hmm It would have been a v4.4 build. Could very well just be coincidence >> I haven’t seen it again. >> >> Either way, I appreciate the info! I’ve been wary of trying THP/KSM for a >> while because of that, but this renews my faith. The clarification about >> THP is also a relief. There’s still so many articles/posts that list these >> ‘gotchas’ regarding VFIO (that have been mostly fixed), it is pretty easy >> to go on unneeded witch hunts =\ >> >> Regarding KSM though… as far as i can tell (in the 4.4 kernel) KSM >> doesn’t use the normal GUP style references (uses get_user_pages_fast >> internally): https://github.com/torvalds/linux/blob/v4.4/mm/ksm.c#L887 >> >> I’m not horribly familiar with KSM itself, but from what I gathered of >> its history, this may be a mechanism it uses to allow shared pages to still >> go to swap etc. >> >> Also afaik KSM, THP and kswapd (@ v4.4) all manipulate the mm >> differently, with KSM and THP having their own unique problems dealing with >> each other. This could also have been some kind of perfect storm, using >> basically all of the memory-mapping technologies modern linux has, at the >> same time… >> >> I’ll need to step through the entire pre and post 4.5 TLB changes and see >> if I anything looks familiar to the current KSM mapping/ref-tracking. >> >> Unfortunately I don’t have any other info at the time- I may try to run >> some more isolated tests, but it was an intermittent issue that required a >> few hours of gaming to flush out… so might take a bit there =\ >> >> On Tue, May 31, 2016 at 3:34 PM Alex Williamson < >> alex.william...@redhat.com> wrote: >> >>> On Tue, 31 May 2016 20:20:58 +0000 >>> Colin Godsey <crgod...@gmail.com> wrote: >>> >>> > I had a few questions regarding general ‘page management’ and VFIO, >>> mostly >>> > related to kernel shared pages. >>> > >>> > I have a host running 2 virtual ‘gaming rigs’ with a single dedicated >>> GPU >>> > each. I had an intermittent problem where when gaming (on the same >>> game) >>> > with both rigs, one would receive graphic artifacts. Specifically I >>> would >>> > see triangle/geometry artifacts which usually indicate corrupt GPU RAM. >>> > >>> > Both cards are so completely different, and different generation, one >>> is >>> > really new, I didn’t believe it was bad VRAM. Graphics drivers to swap >>> > various buffers from system RAM to VRAM so I figured it could also be >>> > something related to system RAM. >>> > >>> > I disabled any kind of… alternative page management I could- swap, KSM, >>> > huge pages etc. and it did fix it. Because the issue only would effect >>> one >>> > machine, and I only observed it when the same game was running on >>> both, I >>> > assumed maybe it was related to KSM. >>> > >>> > *Is there any possible way KSM could interfere with the DMAR in some >>> way >>> > where it tries to share/alter DMA regions?* And broader: what prevents >>> > systems like khugepage, kswap, and ksm from interfering with these >>> regions >>> > in the first place? I’ve read that transparent hugepages can interfere >>> with >>> > VFIO, is it safe to assume that other DMA issues could arise with other >>> > types of page management? >>> >>> What kernel were you running where you saw this? vfio uses >>> get_user_pages to increase the reference count on pages mapped through >>> the iommu. This should prevent both ksm and transparent hugepages from >>> being able to operate on the pages. Kernel v4.5 had a bug (now fixed >>> in v4.5.5) that did not honor the reference, allowing thp (maybe ksm >>> too) to still operate on those pages. So as long as you're not running >>> v4.5.0 through v4.5.4 (or a v4.5-rc), I'm not aware of any issues with >>> page pinning. Thanks, >>> >>> Alex >>> >>
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
