On Tue, Dec 4, 2018 at 11:45 PM Maran Wilson <maran.wil...@oracle.com> wrote: > > On 12/3/2018 8:35 AM, Stefano Garzarella wrote: > > On Mon, Dec 3, 2018 at 4:44 PM Rob Bradford <robert.bradf...@intel.com> > > wrote: > >> Hi Stefano, thanks for capturing all these numbers, > >> > >> On Mon, 2018-12-03 at 15:27 +0100, Stefano Garzarella wrote: > >>> Hi Rob, > >>> I continued to investigate the boot time, and as you suggested I > >>> looked also at qemu-lite 2.11.2 > >>> (https://github.com/kata-containers/qemu) and NEMU "virt" machine. I > >>> did the following tests using the Kata kernel configuration > >>> ( > >>> https://github.com/kata-containers/packaging/blob/master/kernel/configs/x86_64_kata_kvm_4.14.x > >>> ) > >>> > >>> To compare the results with qemu-lite direct kernel load, I added > >>> another tracepoint: > >>> - linux_start_kernel: first entry of the Linux kernel > >>> (start_kernel()) > >>> > >> Great, do you have a set of patches available that all these trace > >> points. It would be great for reproduction. > > For sure! I'm attaching a set of patches for qboot, seabios, ovmf, > > nemu/qemu/qemu-lite and linux 4.14 whit the tracepoints. > > I'm also sharing a python script that I'm using with perf to extract > > the numbers in this way: > > > > $ perf record -a -e kvm:kvm_entry -e kvm:kvm_pio -e > > sched:sched_process_exec -o /tmp/qemu_perf.data & > > $ # start qemu/nemu multiple times > > $ killall perf > > $ perf script -s qemu-perf-script.py -i /tmp/qemu_perf.data > > > >>> As you can see, NEMU is faster to jump to the kernel > >>> (linux_start_kernel) than qemu-lite when uses qboot or seabios with > >>> virt support, but the time to the user space is strangely high, maybe > >>> the kernel configuration that I used is not the best one. > >>> Do you suggest another kernel configuration? > >>> > >> This looks very bad. This isn't the kernel configuration we normally > >> test with in our automated test system but is definitely one we support > >> as part of our partnernship with the Kata team. It's a high priority > >> for me to try and investigate that. Have you saved the kernel messages > >> as they might be helpful? > > Yes, I'm attaching the dmesg output with nemu and qemu. > > > >>> Anyway, I obtained the best boot time with qemu-lite and direct > >>> kernel > >>> load (vmlinux ELF image). I think because the kernel was not > >>> compressed. Indeed, looking to the others test, the kernel > >>> decompression (bzImage) takes about 80 ms (linux_start_kernel - > >>> linux_start_boot). (I'll investigate better) > >>> > >> Yup being able to load an uncompressed kernel is one of the big > >> advantages of qemu-lite. I wonder if we could bring that feature into > >> qemu itself to supplement the existing firmware based kernel loading. > > I think so, I'll try to understand if we can merge the qemu-lite > > direct kernel loading in qemu. > > An attempt was made a long time ago to push the qemu-lite stuff (from > the Intel Clear Containers project) upstream. As I understand it, the > main stumbling block that seemed to derail the effort was that it > involved adding Linux OS specific code to Qemu so that Qemu could do > things like create and populate the zero page that Linux expects when > entering startup_64(). > > That ends up being a lot of very low-level, operating specific knowledge > about Linux that ends up getting baked into Qemu code. And > understandably, a number of folks saw problems with going down a path > like that. > > Since then, we have put together an alternative solution that would > allow Qemu to boot an uncompressed Linux binary via the x86/HVM direct > boot ABI (https://xenbits.xen.org/docs/unstable/misc/pvh.html). The > solution involves first making changes to both the ABI as well as Linux, > and then updating Qemu to take advantage of the updated ABI which is > already supported by both Linux and Free BSD for booting VMs. As such, > Qemu can remain OS agnostic, and just be programmed to the published ABI. > > The canonical definition for the HVM direct boot ABI is in the Xen tree > and we needed to make some minor changes to the ABI definition to allow > KVM guests to also use the same structure and entry point. Those changes > were accepted to the Xen tree already: > https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg00057.html > > The corresponding Linux changes that would allow KVM guests to be booted > via this PVH entry point have already been posted and reviewed: > https://lkml.org/lkml/2018/4/16/1002 > > The final part is the set of Qemu changes to take advantage of the above > and boot a KVM guest via an uncompressed kernel binary using the entry > point defined by the ABI. Liam Merwick will be posting some RFC patches > very soon to allow this.
Hi Maran, thank you very much for the details! I'll take a look to the HVM direct boot ABI. Thanks, Stefano > > Thanks, > -Maran > > > > > Thanks, > > Stefano > > > > > -- Stefano Garzarella Red Hat
