** Project changed: qemu => qemu (Ubuntu)
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1797332
Title:
qemu nested virtualization is not working with Ubuntu16.04
Status in qemu package in Ubuntu:
New
Bug description:
# 1 What am I trying to do ? #
I want to use `libvirt` `qemu/KVM` with **nested virtualization** like
described
in [1] and [2].
**But it does not work with Ubuntu16.04.** It worked some times ago, but not
anymore.
I want 2 levels of virtualization like this:
* L0 – the bare metal host, running KVM on `Ubuntu 16.04`
* L1 – a `Ubuntu 16.04` VM running on L0; also called the "guest hypervisor"
— as it itself is capable of running KVM
* L2 – a `Ubuntu 16.04` VM running on L1, also called the "nested guest"
[1]
https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/
[2] https://www.linux-kvm.org/page/Nested_Guests
My goal is to deploy an `OpenStack` environnement on top of VMs rather than on
bare metal hosts for convenience for a lab experiment. As a result, the
`OpenStack` nodes are L1 VMs. Compute nodes are L1 VMs as well and the VMs
created with `OpenStack` and wich are running on the compute nodes are L2 VMs.
# 2 What is my problem ? #
I can **not** run my 2nd levels of virtualization in 16.04:
* L0 is just fine: running `Ubuntu 16.04.5 LTS`, installed with the `.iso`
image
* L1: I install `libvirt` + `KVM` on L0. I can run VMs like the `Ubuntu16.04`
cloud image on L0.
* L2: I install `libvirt` + `KVM` on L1 as well. But I **can not** run VMs on
L1: I get `kernel panic` or `general protection fault`.
**But if I do the same with Ubuntu18.04** (on the same hardware) instead of
`Ubuntu16.04`, it works without faults.
I don't change the configuration or `virt-install scripts` (other than using
the 18.04 .iso and cloud image).
# 3 My libvirt installation for Ubuntu16.04 #
I install `libvir KVM` in both L0 and L1 using a custom repository [3] from
`OpenStack` team, because their version of libvirt in this repo is newer than
the one on Ubuntu 16.04 official repo and it match the version of `libvirt`
in Ubuntu 18.04.
[3] https://wiki.ubuntu.com/OpenStack/CloudArchive
# 4 hardware and CPU #
CPU is:
> Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
> Intel virt is enable in the bios/uefi.
The rest is standard HDD, standard I/O...
# 5 .iso and cloud image #
I download .iso for L0 bare metal server and cloud image
for L1/L2 VMs from official repository:
Ubuntu 16.04
* http://releases.ubuntu.com/16.04/
* https://cloud-images.ubuntu.com/releases/16.04/release/
Ubuntu 18.04
* http://releases.ubuntu.com/bionic/
* https://cloud-images.ubuntu.com/releases/18.04/release/
# 6 Details #
## Details about L0 Ubuntu 16.04 bare metal host ##
L0 is running `Ubuntu 16.04.5 LTS` installed with the .iso.
**kernel**
```
user@L0:~$ uname -a
Linux L0 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
```
**libvirt version** running on L0
```
user@L0:~$ virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1
```
**qemu version detail**
```
ukvm2@kvm2:~$ qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
```
**KVM acceleration**
```
user@L0:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
```
**nested parameter**
```
user@L0:~$ cat /sys/module/kvm_intel/parameters/nested
Y
```
**number of CPU**
```
user@L0:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
48
```
## Details about a L1 Ubuntu 16.04 VM ##
A VM in L1 (which is running on L0) which is running `Ubuntu 16.04.5 LTS`
installed by a cloud image.
**kernel**
```
user@L1-VM:~$ uname -a
Linux L1 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
```
**libvirt version** running on the L1 VM
```
user@L1-VM:~$ sudo virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1
```
**qemu version detail**
```
user@L1-VM:~$ qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
```
**KVM acceleration**
```
user@L1-VM:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
```
**nested parameter**
```
user@L1-VM:~$ cat /sys/module/kvm_intel/parameters/nested
Y
```
**number of CPU**, which are vCPU given by L0 to the L1 VM
I give 20 vCPU.
```
user@L1-VM:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
20
```
## L1 VM virt-install script parameter ##
If you want to reproduce an L1 VM, I followed this [4]:
```
virt-install \
--connect=qemu:///system \
--name $VMName \
--memory $RAM \
--vcpus $VCPUS \
--cpu host \
--metadata description=$DESCRIPTION \
--os-type linux \
--os-variant ubuntu16.04 \
--disk $DISK_PATH/$VMName.$DISK_FORMAT,size=$DISK_SIZE,bus=virtio \
--disk $CFGIMG_PATH/config_$VMName.$DISK_FORMAT,device=cdrom \
--network bridge=virbr0 \
--graphics none \
--console pty,target_type=serial \
--hvm
```
[4] https://youth2009.org/post/kvm-with-ubuntu-cloud-image/
## Details about a L2 VM ##
I want to create a L2 `Ubuntu 16.04.5 LTS` VM installed by a cloud image VM
within my L1 `KVM` VM. But whatever I do, my L2 VM crash before finishing to
be
instantiated. I get `kernel panic` or `general protection fault`.
Here is the log of an L2 VM after the instanciation failed:
```
user@L1-VM:~$ less /var/log/libvirt/qemu/VMNAME.log
2018-10-11T07:40:45.837151Z qemu-system-x86_64: -chardev pty,id=charserial0:
char device redirected to /dev/pts/1 (label charserial0)
2018-10-11T07:40:45.844279Z qemu-system-x86_64: warning: host doesn't support
requested feature: CPUID.07H:EBX.invpcid [bit 10]
2018-10-11T07:40:45.848532Z qemu-system-x86_64: warning: host doesn't support
requested feature: CPUID.07H:EBX.invpcid [bit 10]
```
If you want to reproduce an L2 VM running on L1, follow [4].
**However** a Cirros OS image can run on a L1 VM !
# 7 Thoughts #
I think this is a bug in either `Ubuntu16.04` or `libvirt`.
All the information are here to reproduce the bug, I think.
If I do the same with `Ubuntu 18.04`, on the same hardware, following the
same
steps but with Ubuntu 18.04 .iso and cloud image, it works.
It works if:
* L0 = Ubuntu18.04 (.iso) + qemu/KVM
* L1 = Ubuntu18.04 (cloud image) + qemu/KVM
* L2 = Ubuntu18.04 (cloud image)
It also works if:
* L0 = Ubuntu18.04 (.iso) + qemu/KVM
* L1 = Ubuntu18.04 (cloud image) + qemu/KVM
* L2 = Ubuntu16.04 (cloud image)
Thank you for your time reading !
--
nico
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions
