On 2019-07-22 21:11:40 -0400, Andrew Cooper wrote: > On 23/07/2019 00:50, Johnson, Ethan wrote: > > (The Xen 4.7 version of the documentation had a little bit > > more detail at https://xenbits.xen.org/docs/4.7-testing/misc/pvh.html, but > > it still doesn't have quite what I'm looking for, and I'm not sure if > > that's for PVHv2 or the older PVHv1.) > > That document was for PVHv1 which is completely obsolete and gone now. > As for these days... > > Xen has two fundamental types of virtualisation. Ring deprivileging > which is what PV guests use, and hardware extensions (Intel VT-x, AMD > SVM) which is what HVM guests use. PVH is identical to HVM in this > regard; what distinguishes PVH is that there is no Qemu providing an > emulation of a IBM-compatible PC behind the scenes. > > PV guests share an address space with Xen. In this regard, they are > very similar to userspace sharing pagetables with the guest kernel. In > our case, we distinguish guest user, guest kernel, and Xen, but the same > basic principles hold. > > HVM guests get an entirely separate address space, either provided by > hardware extensions (Intel EPT, AMD NPT), or via shadow pagetables. > Either way, Xen itself is not a part of this address space, but does > manage it from the outside. > > > 1. Where does the Xen hypervisor itself live in the host's (VMX root) > > virtual and physical address spaces under PVH and HVM? In classic PV it > > seems it lived just below the guest kernel, but that was without extended > > paging separating the host and guest's views of memory. > > Where exactly Xen lives is VMX root mode is of no concern to an HVM/PVH > guest. That said, it is in practice the same as for PV so we can run PV > and HVM/PVH guests at the same time. (Its only recently we've introduce > the ability to selectively compile out PV or HVM support.) > > http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/asm-x86/config.h;h=9ef9d03ca7199ccca416c6ea2d24adaf6dbc4e0f;hb=refs/heads/staging#l116 > describes Xen's virtual memory layout, including the bits applicable to > PV guests. > > Virtually, Xen lives in the first 8Tb of address space of the upper > canonical half, while physically, the only restriction is that it must > be in the bottom 4G of RAM. The exact location is machine specific, but > defaults to highest (sufficiently sized) block of RAM below the 4G boundary. > > > 2. Does Xen also live somewhere within the guest's (VMX non-root) view of > > physical memory as provided by extended paging, in PVH mode? (I'm guessing > > it wouldn't for HVM, but I'm wondering about PVH because it did for classic > > PV.) > > No. For PVH (technically v2, but we don't call it that any more), think > "HVM without Qemu". > > PVHv1 was very misguided, which is why it no longer exists. > > > 3. How are hypercalls handled in PVH/HVM mode? Do they use the VMCALL > > instruction (or something comparable) to initiate a VM exit, or are they > > sometimes handled within VMX non-root mode without a VM exit, similar to > > classic PV? > > Ah - this document is one I prepared earlier. To answer your lower > question, our docs are currently terrible, and I'm trying to improve > things. This is the first part of what I hope will be a document that > would eventually cover all of your questions. Feedback welcome, > including if there is anything unclear in the few bits which currently > exist. > > http://xenbits.xen.org/docs/sphinx-unstable/guest-guide/x86/hypercall-abi.html > > > > 4. How does Xen access the guest's memory when it needs to examine or > > modify it? Does it just access the physical frames through a direct map in > > VMX root mode, or are guests' views of physical memory somehow represented > > somewhere in the host's (hypervisor's) virtual address space? > > Xen has a directmap of host memory, but we're looking to reduce the use > of that as a defence against speculative sidechannels. Xen does not > have a 1:1 directmap of guest physical address space (i.e. we don't have > anything like Qemu/KVM has). > > All physical accesses into guest memory start by walking the P2M (the > EPT/NPT or Shadow pagetables) to find the target data. For accesses > which are based on virtual addresses rather than guest physical, we > perform a pagewalk of the guests pagetables to arrive at the target. > > For emulated MMIO, frames may not be present, at which case special > handling kicks in, such as forwarding the IO request to Qemu, or > terminating it with default x86 behaviour (read ~0, write discard). > > > 5. How do PVH guests learn what their physical memory layout looks like? > > Which regions of memory do they use to communicate with Xen? How is this > > process different for HVM (non-PVH) guests? > > By default, the E820 table passed at boot in the PVH start_info > structure. There is also a hypercall which can be used to obtain a copy > of the E820 table which the toolstack produced. > > > 6. How does dom0 factor into all this? Does it have any direct access to > > domUs' physical memory pages, or does it have to ask Xen to do that on its > > behalf with a hypercall? > > Dom0 is just a plain VM, with some default permissions. In this regard, > it is just like a root process running with more privilege than regular > userspace. > > One concept Xen has is that of a "foreign memory map" which allows dom0 > (or other sufficiently privileged domains) to map guest memory. This is > used for all kinds of tasks, from booting the guest in the first place, > to live migration, and for Qemu to emulate DMA access for an HVM guest. > > > Thanks in advance for any help anyone can offer. If there are any good > > resources out there (books? blog posts?) for understanding the big-picture > > structure of the Xen architecture and codebase as it exists in its modern > > form (i.e., in the PVH/HVM era), those would be especially appreciated. > > I am trying to make things better. For now, asking here is probably > your best bet. > > In some copious free time, I'll try translating this into the guest > guide which was linked above. > > I hope this is all clear and easy to follow. > > ~Andrew
Thanks for taking the time to write out all these great answers - this is exactly what I was looking for! (Apologies for not responding sooner; for whatever reason, your reply didn't make it to my inbox and I had to find it on the list's web archives. I suspect one of my own inbox filters I set up to winnow this high-traffic list is to blame...) Sincerely, Ethan Johnson -- Ethan J. Johnson Computer Science PhD student, Systems group, University of Rochester ejohn...@cs.rochester.edu ethanjohn...@acm.org PGP public key available from public directory or on request _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
