Public bug reported: Description =========== When creating an Ubuntu Xenial instance from the official cloud image (https://cloud-images.ubuntu.com/xenial/current/), I noticed that sometimes it would boot just fine, sometimes it would crash almost immediately with a segmentation fault. After some investigation, I noticed that Xenial guest were crashing when running on my more recent hypervisors (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz), while they would run just fine on my older hypervisors (Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz).
The difference between those two CPUs is that the newer one supports AVX2, while the older one supports AVX only. I realize that this bug is actually a manifestation of #1524069, and has more to do with Libvirt/QEMU than Nova, but it also looks like Nova could be made to configure Libvirt in a way that would work around this bug, which would be great. Steps to reproduce ================== * openstack server create --image ubuntu-1604-xenial --flavor m1.small --nic net-id=c418b807-55e0-4bcf-b31d-6206594a5311 foo Expected result =============== Have a functioning Xenial instance. Actual result ============= The instance crashes shortly after loading raid6: Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1 [ 3.554621] md: multipath personality registered for level -4 [ 3.559487] md: raid0 personality registered for level 0 [ 3.564545] md: raid1 personality registered for level 1 [ 3.639509] raid6: sse2x1 gen() 6920 MB/s [ 3.707518] raid6: sse2x1 xor() 5781 MB/s [ 3.775508] raid6: sse2x2 gen() 9630 MB/s [ 3.843506] raid6: sse2x2 xor() 5438 MB/s [ 3.911506] raid6: sse2x4 gen() 11722 MB/s [ 3.979503] raid6: sse2x4 xor() 7960 MB/s [ 3.983515] invalid opcode: 0000 [#1] SMP [ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy [ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu [ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014 [ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000 [ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246 [ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae [ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012 [ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd [ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000 [ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012 [ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0 [ 3.999840] Stack: [ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0 [ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116 [ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600 [ 4.004020] Call Trace: [ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq] [ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000 [ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200 [ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30 [ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30 [ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0 [ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf [ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00 [ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60 [ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80 [ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0 [ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10 [ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84 [ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 4.023317] RSP <ffff8800b9a4bb78> [ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]--- Segmentation fault Environment =========== 1. I'm running OpenStack Liberty: ii nova-common 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - common files ii nova-compute 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base ii nova-compute-kvm 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM) ii nova-compute-libvirt 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support ii python-nova 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute Python libraries ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API 2. I'm using Libvirt (1.2.16-2ubuntu11.15.10.3~cloud0) + KVM (1:2.3 +dfsg-5ubuntu9.2~cloud0) 3. I'm using Ceph 9.2.1-1trusty as a storage backend 4. I'm using Neutron with Openvswitch for networking Logs & Configs ============== Here's the full console log for an instance that crashes: [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic 4.4.6) [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] x86/fpu: Legacy x87 FPU detected. [ 0.000000] x86/fpu: Using 'lazy' FPU context switches. [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] e820: last_pfn = 0xbffe0 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f6470-0x000f647f] mapped at [ffff8800000f6470] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x371c6000-0x378dafff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F6240 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1672 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE0BDA 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 000B9A (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: SSDT 0x00000000BFFE0C4E 0009AC (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 0x00000000BFFE15FA 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13fff9000-0x13fffdfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 1:3fff5001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 1523727129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffdffff] [ 0.000000] node 0: [mem 0x0000000100000000-0x000000013fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffe0000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88013fc00000 s98008 r8192 d28968 u2097152 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13fc0d940 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032041 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Memory: 4035988K/4193784K available (8356K kernel code, 1278K rwdata, 3920K rodata, 1476K init, 1292K bss, 157796K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1 [ 0.000000] NR_IRQS:16640 nr_irqs:256 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty1] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] tsc: Detected 2599.996 MHz processor [ 0.203234] Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=10399984) [ 0.205106] pid_max: default: 32768 minimum: 301 [ 0.206149] ACPI: Core revision 20150930 [ 0.208546] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.210119] Security Framework initialized [ 0.211083] Yama: becoming mindful. [ 0.211892] AppArmor: AppArmor initialized [ 0.213706] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.218689] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.220490] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.221766] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.223302] Initializing cgroup subsys io [ 0.224173] Initializing cgroup subsys memory [ 0.225039] Initializing cgroup subsys devices [ 0.225972] Initializing cgroup subsys freezer [ 0.226861] Initializing cgroup subsys net_cls [ 0.227736] Initializing cgroup subsys perf_event [ 0.228640] Initializing cgroup subsys net_prio [ 0.229564] Initializing cgroup subsys hugetlb [ 0.230503] Initializing cgroup subsys pids [ 0.231432] CPU: CPU feature xsave disabled, no CPUID level 0xd [ 0.233625] mce: CPU supports 10 MCE banks [ 0.234535] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.235488] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.250916] Freeing SMP alternatives memory: 28K (ffffffff820b2000 - ffffffff820b9000) [ 0.258620] ftrace: allocating 31878 entries in 125 pages [ 0.292291] smpboot: Max logical packages: 1 [ 0.292979] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.294062] x2apic enabled [ 0.294775] Switched APIC routing to physical x2apic. [ 0.296242] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.297081] smpboot: CPU0: Intel Core i7 9xx (Nehalem Class Core i7) (family: 0x6, model: 0x1a, stepping: 0x3) [ 0.298727] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only. [ 0.300782] x86: Booted up 1 node, 1 CPUs [ 0.301405] smpboot: Total of 1 processors activated (5199.99 BogoMIPS) [ 0.303089] devtmpfs: initialized [ 0.306053] evm: security.selinux [ 0.306590] evm: security.SMACK64 [ 0.308354] evm: security.SMACK64EXEC [ 0.308915] evm: security.SMACK64TRANSMUTE [ 0.309542] evm: security.SMACK64MMAP [ 0.310113] evm: security.ima [ 0.310586] evm: security.capability [ 0.311236] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.312590] pinctrl core: initialized pinctrl subsystem [ 0.313427] RTC time: 5:35:03, date: 05/04/16 [ 0.315133] NET: Registered protocol family 16 [ 0.315908] cpuidle: using governor ladder [ 0.316534] cpuidle: using governor menu [ 0.317108] PCCT header not found. [ 0.317686] ACPI: bus type PCI registered [ 0.318298] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319192] PCI: Using configuration type 1 for base access [ 0.320935] ACPI: Added _OSI(Module Device) [ 0.321524] ACPI: Added _OSI(Processor Device) [ 0.322129] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322761] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324221] ACPI: Interpreter enabled [ 0.324764] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150930/hwxface-580) [ 0.326118] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150930/hwxface-580) [ 0.327447] ACPI: (supports S0 S3 S4 S5) [ 0.328000] ACPI: Using IOAPIC for interrupt routing [ 0.328670] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.331711] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332501] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.333346] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.334219] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335856] acpiphp: Slot [3] registered [ 0.336431] acpiphp: Slot [4] registered [ 0.336997] acpiphp: Slot [5] registered [ 0.337563] acpiphp: Slot [6] registered [ 0.339449] acpiphp: Slot [7] registered [ 0.348585] acpiphp: Slot [8] registered [ 0.349160] acpiphp: Slot [9] registered [ 0.349731] acpiphp: Slot [10] registered [ 0.350315] acpiphp: Slot [11] registered [ 0.350891] acpiphp: Slot [12] registered [ 0.351465] acpiphp: Slot [13] registered [ 0.352053] acpiphp: Slot [14] registered [ 0.352637] acpiphp: Slot [15] registered [ 0.353224] acpiphp: Slot [16] registered [ 0.353799] acpiphp: Slot [17] registered [ 0.354382] acpiphp: Slot [18] registered [ 0.354956] acpiphp: Slot [19] registered [ 0.355533] acpiphp: Slot [20] registered [ 0.356105] acpiphp: Slot [21] registered [ 0.356686] acpiphp: Slot [22] registered [ 0.357261] acpiphp: Slot [23] registered [ 0.357834] acpiphp: Slot [24] registered [ 0.358500] acpiphp: Slot [25] registered [ 0.359114] acpiphp: Slot [26] registered [ 0.359721] acpiphp: Slot [27] registered [ 0.360294] acpiphp: Slot [28] registered [ 0.360884] acpiphp: Slot [29] registered [ 0.361463] acpiphp: Slot [30] registered [ 0.362061] acpiphp: Slot [31] registered [ 0.362640] PCI host bridge to bus 0000:00 [ 0.363218] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.364895] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365913] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.366933] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.377218] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378115] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379563] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.380477] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387873] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388875] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465906] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.467014] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.468064] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.469125] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.471532] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.472585] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.473485] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.474215] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.475270] vgaarb: loaded [ 0.475696] vgaarb: bridge control possible 0000:00:02.0 [ 0.476672] SCSI subsystem initialized [ 0.477271] ACPI: bus type USB registered [ 0.477860] usbcore: registered new interface driver usbfs [ 0.478614] usbcore: registered new interface driver hub [ 0.479309] usbcore: registered new device driver usb [ 0.480158] PCI: Using ACPI for IRQ routing [ 0.480957] NetLabel: Initializing [ 0.481458] NetLabel: domain hash size = 128 [ 0.482057] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.482716] NetLabel: unlabeled traffic allowed by default [ 0.483498] clocksource: Switched to clocksource kvm-clock [ 0.490934] AppArmor: AppArmor Filesystem Enabled [ 0.491635] pnp: PnP ACPI init [ 0.492436] pnp: PnP ACPI: found 6 devices [ 0.500252] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.501486] NET: Registered protocol family 2 [ 0.502247] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.503283] TCP bind hash table entries: 32768 (order: 7, 524288 bytes) [ 0.504146] TCP: Hash tables configured (established 32768 bind 32768) [ 0.504963] UDP hash table entries: 2048 (order: 4, 65536 bytes) [ 0.505719] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes) [ 0.506557] NET: Registered protocol family 1 [ 0.507157] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.507914] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.508646] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.510002] Trying to unpack rootfs image as initramfs... [ 1.717098] Freeing initrd memory: 7252K (ffff8800371c6000 - ffff8800378db000) [ 1.718412] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.719192] software IO TLB [mem 0xbbfe0000-0xbffe0000] (64MB) mapped at [ffff8800bbfe0000-ffff8800bffdffff] [ 1.725831] Scanning for low memory corruption every 60 seconds [ 1.731157] futex hash table entries: 256 (order: 2, 16384 bytes) [ 1.731891] audit: initializing netlink subsys (disabled) [ 1.732583] audit: type=2000 audit(1462340105.446:1): initialized [ 1.733541] Initialise system trusted keyring [ 1.734188] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 1.734913] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.736700] zbud: loaded [ 1.737254] VFS: Disk quotas dquot_6.6.0 [ 1.737797] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.738984] fuse init (API version 7.23) [ 1.739650] Key type big_key registered [ 1.740289] Key type asymmetric registered [ 1.740823] Asymmetric key parser 'x509' registered [ 1.741422] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 1.742418] io scheduler noop registered [ 1.742951] io scheduler deadline registered (default) [ 1.743580] io scheduler cfq registered [ 1.744138] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.744813] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.745653] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.746619] ACPI: Power Button [PWRF] [ 1.747193] GHES: HEST is not enabled! [ 1.763884] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.764640] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 1.782055] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 1.782747] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.799998] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.800665] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.817293] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 [ 1.817943] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver [ 1.819651] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.842639] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 1.866310] 00:05: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 1.887604] Linux agpgart interface v0.103 [ 1.889620] brd: module loaded [ 1.890749] loop: module loaded [ 1.908069] vda: vda1 [ 1.909586] scsi host0: ata_piix [ 1.910066] scsi host1: ata_piix [ 1.910562] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0e0 irq 14 [ 1.911256] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0e8 irq 15 [ 1.912214] libphy: Fixed MDIO Bus: probed [ 1.912753] tun: Universal TUN/TAP device driver, 1.6 [ 1.913312] tun: (C) 1999-2004 Max Krasnyansky <[email protected]> [ 1.930649] PPP generic driver version 2.4.2 [ 1.931288] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.932010] ehci-pci: EHCI PCI platform driver [ 1.932597] ehci-platform: EHCI generic platform driver [ 1.933234] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.933950] ohci-pci: OHCI PCI platform driver [ 1.934539] ohci-platform: OHCI generic platform driver [ 1.935191] uhci_hcd: USB Universal Host Controller Interface driver [ 1.952374] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.953034] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.953952] uhci_hcd 0000:00:01.2: detected 2 ports [ 1.954651] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040 [ 1.955379] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.956168] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.957139] usb usb1: Product: UHCI Host Controller [ 1.957768] usb usb1: Manufacturer: Linux 4.4.0-21-generic uhci_hcd [ 1.958543] usb usb1: SerialNumber: 0000:00:01.2 [ 1.959264] hub 1-0:1.0: USB hub found [ 1.959936] hub 1-0:1.0: 2 ports detected [ 1.960616] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.962228] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.962900] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.963656] mousedev: PS/2 mouse device common for all mice [ 1.964524] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 1.965802] rtc_cmos 00:00: RTC can wake from S4 [ 1.966583] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 1.967397] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 1.968119] i2c /dev entries driver [ 1.968702] device-mapper: uevent: version 1.0.3 [ 1.969336] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: [email protected] [ 1.970428] ledtrig-cpu: registered to indicate activity on CPUs [ 1.971346] NET: Registered protocol family 10 [ 1.972135] NET: Registered protocol family 17 [ 1.972732] Key type dns_resolver registered [ 1.973409] microcode: CPU0 sig=0x106a3, pf=0x1, revision=0x1 [ 1.974182] microcode: Microcode Update Driver: v2.01 <[email protected]>, Peter Oruba [ 1.975389] registered taskstats version 1 [ 1.975957] Loading compiled-in X.509 certificates [ 1.977206] Loaded X.509 cert 'Build time autogenerated kernel key: fc7c0e9f152f32eca50ea2d9722926e5127af244' [ 1.978456] zswap: loaded using pool lzo/zbud [ 1.980197] Key type trusted registered [ 1.992720] Key type encrypted registered [ 1.993250] AppArmor: AppArmor sha1 policy hashing enabled [ 1.993898] ima: No TPM chip found, activating TPM-bypass! [ 1.994625] evm: HMAC attrs: 0x1 [ 1.995271] Magic number: 0:849:566 [ 1.995909] rtc_cmos 00:00: setting system clock to 2016-05-04 05:35:05 UTC (1462340105) [ 1.997035] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 1.997702] EDD information not available. [ 2.073644] Freeing unused kernel memory: 1476K (ffffffff81f41000 - ffffffff820b2000) [ 2.075469] Write protecting the kernel read-only data: 14336k [ 2.077486] Freeing unused kernel memory: 1872K (ffff88000182c000 - ffff880001a00000) [ 2.079999] Freeing unused kernel memory: 176K (ffff880001dd4000 - ffff880001e00000) Loading, please wait... starting version 229 [ 2.095265] random: udevadm urandom read with 2 bits of entropy available [ 2.140733] virtio_net virtio0 ens3: renamed from eth0 [ 2.148962] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.150419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.179873] FDC 0 is a S82078B [ 2.723675] tsc: Refined TSC clocksource calibration: 2600.056 MHz [ 2.725415] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a71592ee, max_idle_ns: 440795334892 ns Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1 [ 3.554621] md: multipath personality registered for level -4 [ 3.559487] md: raid0 personality registered for level 0 [ 3.564545] md: raid1 personality registered for level 1 [ 3.639509] raid6: sse2x1 gen() 6920 MB/s [ 3.707518] raid6: sse2x1 xor() 5781 MB/s [ 3.775508] raid6: sse2x2 gen() 9630 MB/s [ 3.843506] raid6: sse2x2 xor() 5438 MB/s [ 3.911506] raid6: sse2x4 gen() 11722 MB/s [ 3.979503] raid6: sse2x4 xor() 7960 MB/s [ 3.983515] invalid opcode: 0000 [#1] SMP [ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy [ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu [ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014 [ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000 [ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246 [ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae [ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012 [ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd [ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000 [ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012 [ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0 [ 3.999840] Stack: [ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0 [ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116 [ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600 [ 4.004020] Call Trace: [ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq] [ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000 [ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200 [ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30 [ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30 [ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0 [ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf [ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00 [ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60 [ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80 [ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0 [ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10 [ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84 [ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 4.023317] RSP <ffff8800b9a4bb78> [ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]--- Segmentation fault And here's the corresponding Libvirt XML: <domain type='kvm' id='411'> <name>instance-000030cd</name> <uuid>587a3497-3198-4c19-87c4-ddf449385f71</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="12.0.1"/> <nova:name>test-xenial-segfault</nova:name> <nova:creationTime>2016-05-04 05:39:55</nova:creationTime> <nova:flavor name="m1.small"> <nova:memory>4096</nova:memory> <nova:disk>16</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="07d961b93aed45c0b7989f91c40341ae">username</nova:user> <nova:project uuid="dd4f48116bd942f9b145d20e9f1a11be">project</nova:project> </nova:owner> <nova:root type="image" uuid="4b5b61a5-008c-4424-a899-f6896384a55c"/> </nova:instance> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <shares>1024</shares> </cputune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>OpenStack Foundation</entry> <entry name='product'>OpenStack Nova</entry> <entry name='version'>12.0.1</entry> <entry name='serial'>00000000-0000-0000-0000-0cc47a349290</entry> <entry name='uuid'>587a3497-3198-4c19-87c4-ddf449385f71</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-vivid'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='1' threads='1'/> </cpu> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <auth username='cinder'> <secret type='ceph' uuid='23c33478-af4c-49d0-bb46-9890d6b89084'/> </auth> <source protocol='rbd' name='libvirt/587a3497-3198-4c19-87c4-ddf449385f71_disk'> <host name='172.24.3.17' port='6789'/> <host name='172.24.3.18' port='6789'/> <host name='172.24.3.19' port='6789'/> <host name='172.24.3.25' port='6789'/> <host name='172.24.3.26' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='fa:16:3e:3c:5e:88'/> <source bridge='qbrba81b423-ab'/> <target dev='tapba81b423-ab'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='file'> <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/> <target port='0'/> <alias name='serial0'/> </serial> <serial type='pty'> <source path='/dev/pts/34'/> <target port='1'/> <alias name='serial1'/> </serial> <console type='file'> <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='pty'> <source path='/dev/pts/35'/> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' port='5915' autoport='yes' listen='172.24.1.35' keymap='fr-ch'> <listen type='address' address='172.24.1.35'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <stats period='10'/> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</label> <imagelabel>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</imagelabel> </seclabel> </domain> ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1578079 Title: Ubuntu Xenial guests segfault on hosts with CPUs supporting AVX2 Status in OpenStack Compute (nova): New Bug description: Description =========== When creating an Ubuntu Xenial instance from the official cloud image (https://cloud-images.ubuntu.com/xenial/current/), I noticed that sometimes it would boot just fine, sometimes it would crash almost immediately with a segmentation fault. After some investigation, I noticed that Xenial guest were crashing when running on my more recent hypervisors (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz), while they would run just fine on my older hypervisors (Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz). The difference between those two CPUs is that the newer one supports AVX2, while the older one supports AVX only. I realize that this bug is actually a manifestation of #1524069, and has more to do with Libvirt/QEMU than Nova, but it also looks like Nova could be made to configure Libvirt in a way that would work around this bug, which would be great. Steps to reproduce ================== * openstack server create --image ubuntu-1604-xenial --flavor m1.small --nic net-id=c418b807-55e0-4bcf-b31d-6206594a5311 foo Expected result =============== Have a functioning Xenial instance. Actual result ============= The instance crashes shortly after loading raid6: Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1 [ 3.554621] md: multipath personality registered for level -4 [ 3.559487] md: raid0 personality registered for level 0 [ 3.564545] md: raid1 personality registered for level 1 [ 3.639509] raid6: sse2x1 gen() 6920 MB/s [ 3.707518] raid6: sse2x1 xor() 5781 MB/s [ 3.775508] raid6: sse2x2 gen() 9630 MB/s [ 3.843506] raid6: sse2x2 xor() 5438 MB/s [ 3.911506] raid6: sse2x4 gen() 11722 MB/s [ 3.979503] raid6: sse2x4 xor() 7960 MB/s [ 3.983515] invalid opcode: 0000 [#1] SMP [ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy [ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu [ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014 [ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000 [ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246 [ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae [ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012 [ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd [ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000 [ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012 [ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0 [ 3.999840] Stack: [ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0 [ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116 [ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600 [ 4.004020] Call Trace: [ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq] [ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000 [ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200 [ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30 [ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30 [ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0 [ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf [ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00 [ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60 [ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80 [ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0 [ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10 [ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84 [ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 4.023317] RSP <ffff8800b9a4bb78> [ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]--- Segmentation fault Environment =========== 1. I'm running OpenStack Liberty: ii nova-common 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - common files ii nova-compute 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base ii nova-compute-kvm 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM) ii nova-compute-libvirt 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support ii python-nova 2:12.0.1-0ubuntu1~cloud0 all OpenStack Compute Python libraries ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API 2. I'm using Libvirt (1.2.16-2ubuntu11.15.10.3~cloud0) + KVM (1:2.3 +dfsg-5ubuntu9.2~cloud0) 3. I'm using Ceph 9.2.1-1trusty as a storage backend 4. I'm using Neutron with Openvswitch for networking Logs & Configs ============== Here's the full console log for an instance that crashes: [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic 4.4.6) [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] x86/fpu: Legacy x87 FPU detected. [ 0.000000] x86/fpu: Using 'lazy' FPU context switches. [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] e820: last_pfn = 0xbffe0 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f6470-0x000f647f] mapped at [ffff8800000f6470] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x371c6000-0x378dafff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F6240 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1672 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE0BDA 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 000B9A (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: SSDT 0x00000000BFFE0C4E 0009AC (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 0x00000000BFFE15FA 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13fff9000-0x13fffdfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 1:3fff5001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 1523727129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffdffff] [ 0.000000] node 0: [mem 0x0000000100000000-0x000000013fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffe0000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88013fc00000 s98008 r8192 d28968 u2097152 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13fc0d940 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032041 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Memory: 4035988K/4193784K available (8356K kernel code, 1278K rwdata, 3920K rodata, 1476K init, 1292K bss, 157796K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1 [ 0.000000] NR_IRQS:16640 nr_irqs:256 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty1] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] tsc: Detected 2599.996 MHz processor [ 0.203234] Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=10399984) [ 0.205106] pid_max: default: 32768 minimum: 301 [ 0.206149] ACPI: Core revision 20150930 [ 0.208546] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.210119] Security Framework initialized [ 0.211083] Yama: becoming mindful. [ 0.211892] AppArmor: AppArmor initialized [ 0.213706] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.218689] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.220490] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.221766] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.223302] Initializing cgroup subsys io [ 0.224173] Initializing cgroup subsys memory [ 0.225039] Initializing cgroup subsys devices [ 0.225972] Initializing cgroup subsys freezer [ 0.226861] Initializing cgroup subsys net_cls [ 0.227736] Initializing cgroup subsys perf_event [ 0.228640] Initializing cgroup subsys net_prio [ 0.229564] Initializing cgroup subsys hugetlb [ 0.230503] Initializing cgroup subsys pids [ 0.231432] CPU: CPU feature xsave disabled, no CPUID level 0xd [ 0.233625] mce: CPU supports 10 MCE banks [ 0.234535] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.235488] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.250916] Freeing SMP alternatives memory: 28K (ffffffff820b2000 - ffffffff820b9000) [ 0.258620] ftrace: allocating 31878 entries in 125 pages [ 0.292291] smpboot: Max logical packages: 1 [ 0.292979] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.294062] x2apic enabled [ 0.294775] Switched APIC routing to physical x2apic. [ 0.296242] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.297081] smpboot: CPU0: Intel Core i7 9xx (Nehalem Class Core i7) (family: 0x6, model: 0x1a, stepping: 0x3) [ 0.298727] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only. [ 0.300782] x86: Booted up 1 node, 1 CPUs [ 0.301405] smpboot: Total of 1 processors activated (5199.99 BogoMIPS) [ 0.303089] devtmpfs: initialized [ 0.306053] evm: security.selinux [ 0.306590] evm: security.SMACK64 [ 0.308354] evm: security.SMACK64EXEC [ 0.308915] evm: security.SMACK64TRANSMUTE [ 0.309542] evm: security.SMACK64MMAP [ 0.310113] evm: security.ima [ 0.310586] evm: security.capability [ 0.311236] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.312590] pinctrl core: initialized pinctrl subsystem [ 0.313427] RTC time: 5:35:03, date: 05/04/16 [ 0.315133] NET: Registered protocol family 16 [ 0.315908] cpuidle: using governor ladder [ 0.316534] cpuidle: using governor menu [ 0.317108] PCCT header not found. [ 0.317686] ACPI: bus type PCI registered [ 0.318298] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319192] PCI: Using configuration type 1 for base access [ 0.320935] ACPI: Added _OSI(Module Device) [ 0.321524] ACPI: Added _OSI(Processor Device) [ 0.322129] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322761] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324221] ACPI: Interpreter enabled [ 0.324764] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150930/hwxface-580) [ 0.326118] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150930/hwxface-580) [ 0.327447] ACPI: (supports S0 S3 S4 S5) [ 0.328000] ACPI: Using IOAPIC for interrupt routing [ 0.328670] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.331711] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332501] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.333346] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.334219] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335856] acpiphp: Slot [3] registered [ 0.336431] acpiphp: Slot [4] registered [ 0.336997] acpiphp: Slot [5] registered [ 0.337563] acpiphp: Slot [6] registered [ 0.339449] acpiphp: Slot [7] registered [ 0.348585] acpiphp: Slot [8] registered [ 0.349160] acpiphp: Slot [9] registered [ 0.349731] acpiphp: Slot [10] registered [ 0.350315] acpiphp: Slot [11] registered [ 0.350891] acpiphp: Slot [12] registered [ 0.351465] acpiphp: Slot [13] registered [ 0.352053] acpiphp: Slot [14] registered [ 0.352637] acpiphp: Slot [15] registered [ 0.353224] acpiphp: Slot [16] registered [ 0.353799] acpiphp: Slot [17] registered [ 0.354382] acpiphp: Slot [18] registered [ 0.354956] acpiphp: Slot [19] registered [ 0.355533] acpiphp: Slot [20] registered [ 0.356105] acpiphp: Slot [21] registered [ 0.356686] acpiphp: Slot [22] registered [ 0.357261] acpiphp: Slot [23] registered [ 0.357834] acpiphp: Slot [24] registered [ 0.358500] acpiphp: Slot [25] registered [ 0.359114] acpiphp: Slot [26] registered [ 0.359721] acpiphp: Slot [27] registered [ 0.360294] acpiphp: Slot [28] registered [ 0.360884] acpiphp: Slot [29] registered [ 0.361463] acpiphp: Slot [30] registered [ 0.362061] acpiphp: Slot [31] registered [ 0.362640] PCI host bridge to bus 0000:00 [ 0.363218] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.364895] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365913] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.366933] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.377218] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378115] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379563] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.380477] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387873] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388875] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465906] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.467014] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.468064] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.469125] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.471532] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.472585] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.473485] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.474215] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.475270] vgaarb: loaded [ 0.475696] vgaarb: bridge control possible 0000:00:02.0 [ 0.476672] SCSI subsystem initialized [ 0.477271] ACPI: bus type USB registered [ 0.477860] usbcore: registered new interface driver usbfs [ 0.478614] usbcore: registered new interface driver hub [ 0.479309] usbcore: registered new device driver usb [ 0.480158] PCI: Using ACPI for IRQ routing [ 0.480957] NetLabel: Initializing [ 0.481458] NetLabel: domain hash size = 128 [ 0.482057] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.482716] NetLabel: unlabeled traffic allowed by default [ 0.483498] clocksource: Switched to clocksource kvm-clock [ 0.490934] AppArmor: AppArmor Filesystem Enabled [ 0.491635] pnp: PnP ACPI init [ 0.492436] pnp: PnP ACPI: found 6 devices [ 0.500252] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.501486] NET: Registered protocol family 2 [ 0.502247] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.503283] TCP bind hash table entries: 32768 (order: 7, 524288 bytes) [ 0.504146] TCP: Hash tables configured (established 32768 bind 32768) [ 0.504963] UDP hash table entries: 2048 (order: 4, 65536 bytes) [ 0.505719] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes) [ 0.506557] NET: Registered protocol family 1 [ 0.507157] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.507914] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.508646] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.510002] Trying to unpack rootfs image as initramfs... [ 1.717098] Freeing initrd memory: 7252K (ffff8800371c6000 - ffff8800378db000) [ 1.718412] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.719192] software IO TLB [mem 0xbbfe0000-0xbffe0000] (64MB) mapped at [ffff8800bbfe0000-ffff8800bffdffff] [ 1.725831] Scanning for low memory corruption every 60 seconds [ 1.731157] futex hash table entries: 256 (order: 2, 16384 bytes) [ 1.731891] audit: initializing netlink subsys (disabled) [ 1.732583] audit: type=2000 audit(1462340105.446:1): initialized [ 1.733541] Initialise system trusted keyring [ 1.734188] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 1.734913] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.736700] zbud: loaded [ 1.737254] VFS: Disk quotas dquot_6.6.0 [ 1.737797] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.738984] fuse init (API version 7.23) [ 1.739650] Key type big_key registered [ 1.740289] Key type asymmetric registered [ 1.740823] Asymmetric key parser 'x509' registered [ 1.741422] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 1.742418] io scheduler noop registered [ 1.742951] io scheduler deadline registered (default) [ 1.743580] io scheduler cfq registered [ 1.744138] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.744813] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.745653] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.746619] ACPI: Power Button [PWRF] [ 1.747193] GHES: HEST is not enabled! [ 1.763884] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.764640] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 1.782055] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 1.782747] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.799998] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.800665] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.817293] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 [ 1.817943] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver [ 1.819651] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.842639] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 1.866310] 00:05: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 1.887604] Linux agpgart interface v0.103 [ 1.889620] brd: module loaded [ 1.890749] loop: module loaded [ 1.908069] vda: vda1 [ 1.909586] scsi host0: ata_piix [ 1.910066] scsi host1: ata_piix [ 1.910562] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0e0 irq 14 [ 1.911256] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0e8 irq 15 [ 1.912214] libphy: Fixed MDIO Bus: probed [ 1.912753] tun: Universal TUN/TAP device driver, 1.6 [ 1.913312] tun: (C) 1999-2004 Max Krasnyansky <[email protected]> [ 1.930649] PPP generic driver version 2.4.2 [ 1.931288] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.932010] ehci-pci: EHCI PCI platform driver [ 1.932597] ehci-platform: EHCI generic platform driver [ 1.933234] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.933950] ohci-pci: OHCI PCI platform driver [ 1.934539] ohci-platform: OHCI generic platform driver [ 1.935191] uhci_hcd: USB Universal Host Controller Interface driver [ 1.952374] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.953034] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.953952] uhci_hcd 0000:00:01.2: detected 2 ports [ 1.954651] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040 [ 1.955379] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.956168] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.957139] usb usb1: Product: UHCI Host Controller [ 1.957768] usb usb1: Manufacturer: Linux 4.4.0-21-generic uhci_hcd [ 1.958543] usb usb1: SerialNumber: 0000:00:01.2 [ 1.959264] hub 1-0:1.0: USB hub found [ 1.959936] hub 1-0:1.0: 2 ports detected [ 1.960616] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.962228] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.962900] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.963656] mousedev: PS/2 mouse device common for all mice [ 1.964524] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 1.965802] rtc_cmos 00:00: RTC can wake from S4 [ 1.966583] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 1.967397] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 1.968119] i2c /dev entries driver [ 1.968702] device-mapper: uevent: version 1.0.3 [ 1.969336] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: [email protected] [ 1.970428] ledtrig-cpu: registered to indicate activity on CPUs [ 1.971346] NET: Registered protocol family 10 [ 1.972135] NET: Registered protocol family 17 [ 1.972732] Key type dns_resolver registered [ 1.973409] microcode: CPU0 sig=0x106a3, pf=0x1, revision=0x1 [ 1.974182] microcode: Microcode Update Driver: v2.01 <[email protected]>, Peter Oruba [ 1.975389] registered taskstats version 1 [ 1.975957] Loading compiled-in X.509 certificates [ 1.977206] Loaded X.509 cert 'Build time autogenerated kernel key: fc7c0e9f152f32eca50ea2d9722926e5127af244' [ 1.978456] zswap: loaded using pool lzo/zbud [ 1.980197] Key type trusted registered [ 1.992720] Key type encrypted registered [ 1.993250] AppArmor: AppArmor sha1 policy hashing enabled [ 1.993898] ima: No TPM chip found, activating TPM-bypass! [ 1.994625] evm: HMAC attrs: 0x1 [ 1.995271] Magic number: 0:849:566 [ 1.995909] rtc_cmos 00:00: setting system clock to 2016-05-04 05:35:05 UTC (1462340105) [ 1.997035] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 1.997702] EDD information not available. [ 2.073644] Freeing unused kernel memory: 1476K (ffffffff81f41000 - ffffffff820b2000) [ 2.075469] Write protecting the kernel read-only data: 14336k [ 2.077486] Freeing unused kernel memory: 1872K (ffff88000182c000 - ffff880001a00000) [ 2.079999] Freeing unused kernel memory: 176K (ffff880001dd4000 - ffff880001e00000) Loading, please wait... starting version 229 [ 2.095265] random: udevadm urandom read with 2 bits of entropy available [ 2.140733] virtio_net virtio0 ens3: renamed from eth0 [ 2.148962] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.150419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.179873] FDC 0 is a S82078B [ 2.723675] tsc: Refined TSC clocksource calibration: 2600.056 MHz [ 2.725415] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x257a71592ee, max_idle_ns: 440795334892 ns Begin: Loading essential drivers ... [ 3.549857] md: linear personality registered for level -1 [ 3.554621] md: multipath personality registered for level -4 [ 3.559487] md: raid0 personality registered for level 0 [ 3.564545] md: raid1 personality registered for level 1 [ 3.639509] raid6: sse2x1 gen() 6920 MB/s [ 3.707518] raid6: sse2x1 xor() 5781 MB/s [ 3.775508] raid6: sse2x2 gen() 9630 MB/s [ 3.843506] raid6: sse2x2 xor() 5438 MB/s [ 3.911506] raid6: sse2x4 gen() 11722 MB/s [ 3.979503] raid6: sse2x4 xor() 7960 MB/s [ 3.983515] invalid opcode: 0000 [#1] SMP [ 3.984205] Modules linked in: raid6_pq(+) libcrc32c raid1 raid0 multipath linear psmouse floppy [ 3.985859] CPU: 0 PID: 230 Comm: modprobe Not tainted 4.4.0-21-generic #37-Ubuntu [ 3.986881] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014 [ 3.988118] task: ffff8800bb1d7080 ti: ffff8800b9a48000 task.ti: ffff8800b9a48000 [ 3.989124] RIP: 0010:[<ffffffffc0061a8d>] [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 3.992121] RSP: 0018:ffff8800b9a4bb78 EFLAGS: 00010246 [ 3.992808] RAX: 0000000000000000 RBX: ffff8800b9a4bbc8 RCX: 00000000fffedeae [ 3.993669] RDX: 0000000000000080 RSI: 0000000000001000 RDI: 0000000000000012 [ 3.994541] RBP: ffff8800b9a4bba8 R08: 0000000000000000 R09: 00000000000001bd [ 3.995446] R10: 00000000fffede9d R11: 00000000000001bd R12: 0000000000001000 [ 3.996307] R13: ffff8800bb022000 R14: ffff8800bb023000 R15: 0000000000000012 [ 3.997167] FS: 00007fa2699f6700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 3.998245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.998974] CR2: 000056292fa0b008 CR3: 00000000b9a19000 CR4: 00000000001006f0 [ 3.999840] Stack: [ 4.000204] 0000000000000080 ffffffffc00747a0 0000000000000001 0000000000003fb0 [ 4.001439] ffffffffc0062238 ffff8800bb022000 ffff8800b9a4bc88 ffffffffc0079116 [ 4.002684] 00000000fffedeae 0000000000002ee4 ffffffffc0064600 ffffffffc0065600 [ 4.004020] Call Trace: [ 4.004429] [<ffffffffc0079116>] init_module+0x116/0x1000 [raid6_pq] [ 4.005243] [<ffffffffc0079000>] ? 0xffffffffc0079000 [ 4.005917] [<ffffffff81002123>] do_one_initcall+0xb3/0x200 [ 4.006651] [<ffffffff818205f8>] ? preempt_schedule_common+0x18/0x30 [ 4.007451] [<ffffffff8182062c>] ? _cond_resched+0x1c/0x30 [ 4.008214] [<ffffffff811eaeb3>] ? kmem_cache_alloc_trace+0x183/0x1f0 [ 4.009021] [<ffffffff8118c163>] do_init_module+0x5f/0x1cf [ 4.009734] [<ffffffff81109df7>] load_module+0x1667/0x1c00 [ 4.010458] [<ffffffff811063a0>] ? __symbol_put+0x60/0x60 [ 4.011163] [<ffffffff812126b0>] ? kernel_read+0x50/0x80 [ 4.011861] [<ffffffff8110a5d4>] SYSC_finit_module+0xb4/0xe0 [ 4.012590] [<ffffffff8110a61e>] SyS_finit_module+0xe/0x10 [ 4.013302] [<ffffffff818244f2>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 4.014104] Code: 55 41 54 53 48 89 d3 48 8d 14 c5 00 00 00 00 41 89 ff 49 89 f4 48 83 ec 08 4c 8b 2c c3 4c 8b 74 13 08 48 89 55 d0 e8 53 83 fd c0 <c5> fd 6f 05 8b 2e 01 00 c5 e5 ef db 4d 85 e4 48 8b 55 d0 0f 84 [ 4.020068] RIP [<ffffffffc0061a8d>] raid6_avx21_gen_syndrome+0x3d/0x120 [raid6_pq] [ 4.023317] RSP <ffff8800b9a4bb78> [ 4.023847] ---[ end trace b3853dc6e5fc1f8f ]--- Segmentation fault And here's the corresponding Libvirt XML: <domain type='kvm' id='411'> <name>instance-000030cd</name> <uuid>587a3497-3198-4c19-87c4-ddf449385f71</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="12.0.1"/> <nova:name>test-xenial-segfault</nova:name> <nova:creationTime>2016-05-04 05:39:55</nova:creationTime> <nova:flavor name="m1.small"> <nova:memory>4096</nova:memory> <nova:disk>16</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="07d961b93aed45c0b7989f91c40341ae">username</nova:user> <nova:project uuid="dd4f48116bd942f9b145d20e9f1a11be">project</nova:project> </nova:owner> <nova:root type="image" uuid="4b5b61a5-008c-4424-a899-f6896384a55c"/> </nova:instance> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <shares>1024</shares> </cputune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>OpenStack Foundation</entry> <entry name='product'>OpenStack Nova</entry> <entry name='version'>12.0.1</entry> <entry name='serial'>00000000-0000-0000-0000-0cc47a349290</entry> <entry name='uuid'>587a3497-3198-4c19-87c4-ddf449385f71</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-vivid'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='1' threads='1'/> </cpu> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <auth username='cinder'> <secret type='ceph' uuid='23c33478-af4c-49d0-bb46-9890d6b89084'/> </auth> <source protocol='rbd' name='libvirt/587a3497-3198-4c19-87c4-ddf449385f71_disk'> <host name='172.24.3.17' port='6789'/> <host name='172.24.3.18' port='6789'/> <host name='172.24.3.19' port='6789'/> <host name='172.24.3.25' port='6789'/> <host name='172.24.3.26' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='fa:16:3e:3c:5e:88'/> <source bridge='qbrba81b423-ab'/> <target dev='tapba81b423-ab'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='file'> <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/> <target port='0'/> <alias name='serial0'/> </serial> <serial type='pty'> <source path='/dev/pts/34'/> <target port='1'/> <alias name='serial1'/> </serial> <console type='file'> <source path='/var/lib/nova/instances/587a3497-3198-4c19-87c4-ddf449385f71/console.log'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='pty'> <source path='/dev/pts/35'/> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' port='5915' autoport='yes' listen='172.24.1.35' keymap='fr-ch'> <listen type='address' address='172.24.1.35'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <stats period='10'/> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</label> <imagelabel>libvirt-587a3497-3198-4c19-87c4-ddf449385f71</imagelabel> </seclabel> </domain> To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1578079/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
