* Christian Borntraeger (borntrae...@de.ibm.com) wrote: > On 04/27/2017 03:47 PM, Andrea Arcangeli wrote: > > On Thu, Apr 27, 2017 at 08:44:03AM +0200, Christian Borntraeger wrote: > >> I have started instrumenting the kernel. I can see a set_pte_at for this > >> address > >> and I see an (to be understood) invalidation shortly after that which > >> explains > >> why I get a fault. > > > > Sounds great that you can see an invalidation shortly after, that is > > the real source of the problem. Can you get a stack trace of such > > invalidation? > > > > Thanks! > > Andrea > > > > Finally got it. I had a test module in that guest, which triggered a storage > key > operation. Normally we no longer use the storage keys in Linux. Therefore KVM > disables storage key support and intercepts all storage key instructions to > enable > the support for that lazily.This makes paging easier and faster to not worry > about those. > When we enable storage keys, we must not use shared pages as the storage key > is a property of the physical page frame (and not of the virtual page). > Therefore, this enablement makes mm_forbids_zeropage return true and removes > all existing zero pages. > (see commit 2faee8ff9dc6f4bfe46f6d2d110add858140fb20 > s390/mm: prevent and break zero page mappings in case of storage keys) > In this case it was called while migrating the storage keys (via kvm ioctl) > resulting in zero page mappings going away. (see qemu hw/s390x/s390-skeys.c) > > > Apr 28 14:48:43 s38lp08 kernel: ([<000000000011218a>] show_trace+0x62/0x78) > Apr 28 14:48:43 s38lp08 kernel: [<0000000000112278>] show_stack+0x68/0xe0 > Apr 28 14:48:43 s38lp08 kernel: [<000000000066f82e>] dump_stack+0x7e/0xb0 > Apr 28 14:48:43 s38lp08 kernel: [<0000000000123b2c>] > ptep_xchg_direct+0x254/0x288 > Apr 28 14:48:43 s38lp08 kernel: [<0000000000127cfe>] > __s390_enable_skey+0x76/0xa0 > Apr 28 14:48:43 s38lp08 kernel: [<00000000002e5278>] > __walk_page_range+0x270/0x500 > Apr 28 14:48:43 s38lp08 kernel: [<00000000002e5592>] > walk_page_range+0x8a/0x148 > Apr 28 14:48:43 s38lp08 kernel: [<0000000000127bc6>] > s390_enable_skey+0x116/0x140 > Apr 28 14:48:43 s38lp08 kernel: [<000000000013fd92>] > kvm_arch_vm_ioctl+0x11ea/0x1c70 > Apr 28 14:48:43 s38lp08 kernel: [<0000000000131aa2>] kvm_vm_ioctl+0xca/0x710 > Apr 28 14:48:43 s38lp08 kernel: [<00000000003460e8>] do_vfs_ioctl+0xa8/0x608 > Apr 28 14:48:43 s38lp08 kernel: [<00000000003466ec>] SyS_ioctl+0xa4/0xb8 > Apr 28 14:48:43 s38lp08 kernel: [<0000000000923460>] system_call+0xc4/0x23c > > As a result a userfault on this virtual address will indeed go back to QEMU > and asks again for that page. And then QEMU "oh I have that page already > transferred" > (even if it was detected as zero page and just faulted in by reading from it) > So I will not write it again. > > Several options: > - let postcopy not discard a page, even it if must already be there (patch > from David)
Yes so that patch was forcing the population of zero pages when received; now that might cause a lot of memory consumption - so it's not a great idea if we can avoid it. > - change s390-skeys to register_savevm_live and do the skey enablement > very early (but this will be impossible for incoming data from old versions) > - let kernel s390_enable_skey actually fault in (might show big memory > consumption) > - let qemu hw/s390x/s390-skeys.c tell the migration code that pages might need > retransmissions Note that it would have to do that on the source side prior to/or in the discard phase of postcopy entry; any later and the source will ignore those requests (which is not easy to fix - because it's a safeguard against it overwriting new data on the destination by old data in a race). Dave > .... > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
