[Xen-devel] dom0 kenrel crashes for openstack + libvirt + libxl
Hi, I am aware that Xen via libvirt is in the group C support for openstack but since I am not able to install xenserver iso at compute machines I have, I have to consider to use xen with libvirt (xcp-xapi is not available for ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow the instruction to install juno in ubuntu 14.04 and it works (I can create instances from openstack GUI - horizon) when I use kvm as the hypervisor at the compute node . However, if I switch to use xen as the hypervisor by installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to create instances. It complained "backend for qemu disk is not ready". I checked and did not find qemu process running in dom0. I could not find /etc/init.d/xencommons either. Then, I compiled and installed xen-4.4 from source code and I got xencommons in /etc/init.d. qemu process is running in dom0. However, the dom0 kernel crashes this time when I try to launch an instance from openstack GUI. I am able to create a xen guest with virt-install from command line, with the following command. "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk u14.04.3.img,size=4,driver_name=qemu --location http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ --network bridge=virbr0" dmesg: [ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: disabled; [ 9443.132818] xvda: xvda1 [ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use! [ 9444.604496] deferring g.e. 0x30 (pfn 0x) [ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use! [ 9444.604502] deferring g.e. 0x31 (pfn 0x) [ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use! [ 9444.604508] deferring g.e. 0x32 (pfn 0x) lots of them [ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use! [ 9444.604721] deferring g.e. 0xe (pfn 0x) [ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use! [ 9444.604725] deferring g.e. 0xd (pfn 0x) [ 9448.325408] [ cut here ] [ 9448.325421] WARNING: CPU: 5 PID: 19902 at /build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x 1a9/0x1b0() [ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF O 3.13.0-33-generic #58-Ubuntu [ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS 2.1.15 09/02/2010 [ 9448.325497] 0009 8802d13d9aa8 8171bd04 [ 9448.325501] 8802d13d9ae0 810676cd 0001 [ 9448.325505] 88030418ffe0 88031d6ab180 0010 8802d13d9af0 [ 9448.325509] Call Trace: [ 9448.325518] [] dump_stack+0x45/0x56 [ 9448.325523] [] warn_slowpath_common+0x7d/0xa0 [ 9448.325526] [] warn_slowpath_null+0x1a/0x20 [ 9448.325530] [] xen_mc_flush+0x1a9/0x1b0 [ 9448.325534] [] xen_set_pud_hyper+0x109/0x110 [ 9448.325538] [] xen_set_pud+0x9b/0xb0 [ 9448.325543] [] __pmd_alloc+0xd6/0x110 [ 9448.325548] [] move_page_tables+0x678/0x720 [ 9448.325552] [] ? vma_adjust+0x337/0x7d0 [ 9448.325557] [] shift_arg_pages+0xbd/0x1b0 [ 9448.325560] [] ? mprotect_fixup+0x151/0x290 [ 9448.325564] [] setup_arg_pages+0x1db/0x200 [ 9448.325570] [] load_elf_binary+0x41c/0xd80 [ 9448.325576] [] ? ima_get_action+0x23/0x30 [ 9448.325579] [] ? process_measurement+0x82/0x2c0 [ 9448.325584] [] search_binary_handler+0x8f/0x1b0 [ 9448.325587] [] do_execve_common.isra.22+0x5a7/0x7e0 [ 9448.325591] [] SyS_execve+0x36/0x50 [ 9448.325596] [] stub_execve+0x69/0xa0 [ 9448.325599] ---[ end trace 53ac16782e751c0a ]--- [ 9448.347994] [ cut here ] [ 9448.348004] WARNING: CPU: 1 PID: 19902 at /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170() more message ignored I sent emails to the openstack mailing list and they gave me a few suggestions. At last, Bob Ball suggested I should probably ask this question on xen-devel mailing list. Any help will be highly appreciated! Thanks, Original discussion on openstack mailing list: http://www.gossamer-threads.com/lists/openstack/dev/42337 -Xing ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] dom0 kenrel crashes for openstack + libvirt + libxl
Hi, Roger is correct: after I applied that patch and reinstalled qemu, modified /etc/init.d/xencommons to use the modified version of qemu to provide disk backend, I finally succeeded to create a xen guest! I noticed that this patch was proposed yesterday: just in-time to solve my problem. :-) root@compute1:/mnt/qemu-2.0.2# virsh --connect=xen:/// Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # list IdName State 1 instance-0028 running virsh # I have been blocked by this problem for almost three weeks and as far as I know, I could not find any online tutorial on setting up xen based compute node for openstack juno. I will document steps I take and share them with the community. Cheers, -Xing On Thu, Nov 13, 2014 at 3:57 AM, Roger Pau Monné wrote: > El 12/11/14 a les 19.41, Xing Lin ha escrit: > > Hi, > > > > I am aware that Xen via libvirt is in the group C support for openstack > but > > since I am not able to install xenserver iso at compute machines I have, > I > > have to consider to use xen with libvirt (xcp-xapi is not available for > > ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow > > the instruction to install juno in ubuntu 14.04 and it works (I can > create > > instances from openstack GUI - horizon) when I use kvm as the hypervisor > at > > the compute node . However, if I switch to use xen as the hypervisor by > > installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to > create > > instances. It complained "backend for qemu disk is not ready". I checked > > and did not find qemu process running in dom0. I could not find > > /etc/init.d/xencommons either. Then, I compiled and installed xen-4.4 > from > > source code and I got xencommons in /etc/init.d. qemu process is running > in > > dom0. However, the dom0 kernel crashes this time when I try to launch an > > instance from openstack GUI. I am able to create a xen guest with > > virt-install from command line, with the following command. > > > > "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk > > u14.04.3.img,size=4,driver_name=qemu --location > > http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ > --network > > bridge=virbr0" > > > > > > dmesg: > > > > [ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent > grants: > > enabled; indirect descriptors: disabled; > > [ 9443.132818] xvda: xvda1 > > [ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use! > > [ 9444.604496] deferring g.e. 0x30 (pfn 0x) > > [ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use! > > [ 9444.604502] deferring g.e. 0x31 (pfn 0x) > > [ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use! > > [ 9444.604508] deferring g.e. 0x32 (pfn 0x) > > lots of them > > [ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use! > > [ 9444.604721] deferring g.e. 0xe (pfn 0x) > > [ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use! > > [ 9444.604725] deferring g.e. 0xd (pfn 0x) > > [ 9448.325408] [ cut here ] > > [ 9448.325421] WARNING: CPU: 5 PID: 19902 at > > /build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x > > 1a9/0x1b0() > > [ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF O > > 3.13.0-33-generic #58-Ubuntu > > [ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS > 2.1.15 > > 09/02/2010 > > [ 9448.325497] 0009 8802d13d9aa8 8171bd04 > > > > [ 9448.325501] 8802d13d9ae0 810676cd > > 0001 > > [ 9448.325505] 88030418ffe0 88031d6ab180 0010 > > 8802d13d9af0 > > [ 9448.325509] Call Trace: > > [ 9448.325518] [] dump_stack+0x45/0x56 > > [ 9448.325523] [] warn_slowpath_common+0x7d/0xa0 > > [ 9448.325526] [] warn_slowpath_null+0x1a/0x20 > > [ 9448.325530] [] xen_mc_flush+0x1a9/0x1b0 > > [ 9448.325534] [] xen_set_pud_hyper+0x109/0x110 > > [ 9448.325538] [] xen_set_pud+0x9b/0xb0 > > [ 9448.325543] [] __pmd_alloc+0xd6/0x110 > > [ 9448.325548] [] move_page_tables+0x678/0x720 > > [ 9448.325552] [] ? vma_adjust+0x337/0x7d0 > > [ 9448.325557] [] shift_arg_pages+0xbd/0x1b0 > > [ 9448.325560] [] ? mprotect_fixup+0x151/
Re: [Xen-devel] dom0 kenrel crashes for openstack + libvirt + libxl
Hi, The wiki page is ready. I am not sure whether I am using the correct format or not. Please let me know if any changes are need. Thanks, http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno -Xing On Fri, Nov 14, 2014 at 2:01 AM, Ian Campbell wrote: > On Thu, 2014-11-13 at 22:54 -0700, Xing Lin wrote: > > I have been blocked by this problem for almost three weeks and as far > > as I know, I could not find any online tutorial on setting up xen > > based compute node for openstack juno. I will document steps I take > > and share them with the community. > > Awesome, thanks! > > Having this in the Xen wiki would be awesome. If you would like write > access to our wiki (which is granted manually due to spammers spoiling > it for everyone...) then please: > > * Create an account using the "create account" link at the > top-right hand side of http://wiki.xen.org/wiki/Main_Page > * Fill in this form giving your chosen username: > > http://xenproject.org/component/content/article/100-misc/145-request-to-be-made-a-wiki-editor.html > > You could alternatively drop me a line privately with the user name for > the second step, but filling in the form might give a quicker response > time due to reaching multiple people in multiple timezones... > > Ian. > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] dom0 kenrel crashes for openstack + libvirt + libxl
Hi Ian, Both of your two points are valid. There is no need to install virt-manager. And the patch to start a qemu process in /etc/init.d/xen seems to be enough for launching instances from horizon. I have updated the wiki page. Please review it at http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno Thanks, Ian! -Xing On Mon, Nov 17, 2014 at 5:39 AM, Ian Campbell wrote: > On Fri, 2014-11-14 at 21:10 -0700, Xing Lin wrote: > > Hi, > > > > > > The wiki page is ready. I am not sure whether I am using the correct > > format or not. Please let me know if any changes are need. Thanks, > > > > > > http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno > > Thanks for this. WRT the need to install virt manager to avoid the > "cannot open shared object file" issue I expect just running "ldconfig" > would have worked instead. > > It would also be good to understand why it is necessary to install from > source. Was it just the lack of the xencommons initscript? Debian and > Ubuntu have their own initscripts and don't reuse the xencommons script. > However it should be fairly easy to add the necessary commands to start > qemu to /etc/init.d/xen instead of rebuilding from source. > > I'd expect just adding to the end of the "start)" section of the script > to work. e.g. > > *) log_end_msg 1; exit ;; > esac > log_end_msg 0 > + /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name > dom0 -nographic -M xenpv -daemonize \ > + -monitor /dev/null -serial /dev/null -parallel /dev/null \ > + -pidfile /var/run/qemu-xen-dom0.pid > ;; > stop) > capability_check > case "$?" in > 0) ;; > > (nb, that's not a real patch, I just typed it into my mail client as is) > > If you can confirm that this works then I can try and get this fixed in > Debian at least. > > Ian. > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] dom0 kenrel crashes for openstack + libvirt + libxl
Hi, I git clone nova-juno from github and searched for 'pygrub'. Here is what i get. .//etc/nova/rootwrap.d/compute.filters:104:# nova/virt/xenapi/vm_utils.py: 'pygrub', '-qn', dev_path .//etc/nova/rootwrap.d/compute.filters:105:pygrub: CommandFilter, pygrub, root .//nova/tests/unit/virt/xenapi/test_xenapi.py:667: self.assertEqual(self.vm['PV_bootloader'], 'pygrub') .//nova/virt/xenapi/vm_utils.py:298:rec['PV_bootloader'] = 'pygrub' It seems that nova does not specify absolute path for pygrub. I checked libvirt source code and found the following definition. .//src/libxl/libxl_conf.h:52:# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub" Searched further for difinition of BINDIR revealed the followings. Xing:libvirt-1.2.2 xing$ grep BINDIR -Rn ./ .//ChangeLog:48645: SBINDIR "/libvirtd" .//ChangeLog:48646: SBINDIR "/libvirtd_dbg" .//build-aux/ltmain.sh:2408: -bindir BINDIRspecify path to binaries directory (for systems where .//gnulib/lib/Makefile.in:2499: echo '#define BINDIR "$(bindir)"'; \ .//gnulib/lib/Makefile.in:2500: echo '#define SBINDIR "$(sbindir)"'; \ .//gnulib/lib/gnulib.mk:303: echo '#define BINDIR "$(bindir)"'; \ .//gnulib/lib/gnulib.mk:304: echo '#define SBINDIR "$(sbindir)"'; \ I guess BINDIR was pointed to /usr/bin. -Xing On Thu, Nov 20, 2014 at 2:30 AM, Ian Campbell wrote: > On Wed, 2014-11-19 at 11:57 -0700, Xing Lin wrote: > > Hi Ian, > > > > > > Both of your two points are valid. There is no need to install > > virt-manager. And the patch to start a qemu process in /etc/init.d/xen > > seems to be enough for launching instances from horizon. I have > > updated the wiki page. Please review it at > > > > http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno > > Looks much better, thanks! > > I think the need to symlink pygrub to /usr/bin should be considered an > openstack bug. I presume nova specifies the absolute path, when it > should just say "pygrub" by default and let the toolstack find the > default one. > > Ian. > > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
