Hello Gabriel,
Thanks for trying our code :) Sorry for the long email but I will try to answer all your questions. We used to support deploying on ubuntu 12.04 however for the last 9 months we have not had capacity to Test our deployment code on Ubuntu. It is good timing however as we are currently working on updating our deployment code to work on Ubuntu. I have just opened a bug to track this here https://bugs.launchpad.net/networking-ovs-dpdk/+bug/1486697 Just as a side note we are currently updateing our documentation to include a short getting started guide for Fedora 21. When we close the above bug we will update the getting started guide we information regarding deployment On ubunutu also. The current fedora guide can be found here https://review.openstack.org/#/c/214156/2 however it will not solve your current issues. Currently we are investigating a number of issues relating to Ubuntu deployment. We are aware of the su command limitation on Ubuntu and will be updating our code to make it compatible. On Ubuntu 12.04 there previously was both a "libvirt-qemu" user and group On 14.04 I belive the "libvirt-qemu" group has changed to "kvm" On Ubuntu 15.04 you should have a new enough Libvirt and qemu installed by default. On 14.04 the kilo branch of the Ubuntu cloud archive will provide the correct qemu and Libvirt versions. In general I recommend Libvirt 1.2.13+ and qemu 2.2+ If you are manually booting with vhost-user via Libvirt in addition to specify the memory backing element <memoryBacking> <hugepages/> </memoryBacking> You also need set the hugepages to be mapped as shared. This can be done via the numa element in the Libvirt xml e.g. <numa> <cell id='0' cpus='0-3' memory='512000' unit='KiB' memAccess='shared'/> </numa> When booting vms with OpenStack this limitation was worked around in our qemu wrapper script for kilo and has be fixed upstream in the liberty nova tree. The following message can be safely ignored. 2015-08-18T22:20:28Z|00011|dpif_netlink|ERR|Generic Netlink family 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably not loaded. As part of the plugin we also unload the kernel module but as it is not required when using dpdk. This error message is issues because our devstack plugin does not currently create the bridges and set their datapath to netdev as a single atomic command. As a result the bridge is initially created with the kernel datapath then updated to the netdev datapath resulting in the above warning. On to your actual question it think the permission denied issue is related to the su issues mentioned previously. Instead of changing "sudo su -g $qemu_group ..." to "sg $qemu_group ..." When starting ovs-vswitchd can you change it as follow "sudo su -g $qemu_group -c..." to "sudo -g $qemu_group ..." The command our service file uses to start ovs is as follows screen -dms ovs-vswitchd sudo su -g $qemu_group -c "umask 002; ${OVS_INSTALL_DIR}/sbin/ovs-vswitchd --dpdk -c $OVS_CORE_MASK -n $OVS_MEM_CHANNELS --proc-type primary --huge-dir $OVS_HUGEPAGE_MOUNT --socket-mem $OVS_SOCKET_MEM $pciAddressWhitelist -- unix:$OVS_DB_SOCKET 2>&1 | tee ${OVS_LOG_DIR}/ovs-vswitchd.log" that is rather log but the important parts are as follows. umask 002 changes the default file creation mode for the ovs-vswitchd process to itself and its group. su -g $qemu_group changes the default group the ovs-vswitchd process runs as part of to the same group as the qemu process. And obviously the sudo causes the ovs-vswitchd process to run as the root user. sudo support setting the group with the -g flag directly so su is not needed. The result of combining these command is that all vhost-user socket created by ovs will be owned by the root user and the Libvirt-qemu users group(previously "libvirt-qemu" now "kvm") Finally to boot a vm successfully via openstack you will need to create a symlink between /usr/var/run/openvswitch and /var/run/openvswitch. The base path for the vhost-user socket is a constant in our mechanism driver as it must be the same on all compute nodes regardless of the operating system. On a side note If anyone knows the correct configure options to specify that binaries should be installed in /usr/bin and all unix socket (ovsdb,ovs-vswictd and vhost-user) should be created in /var/run/openvswitch please let me know. Currently we use ./configure --with-dpdk=${OVS_DPDK_DIR}/${RTE_TARGET} --prefix=/usr However on fedora unix sockets are created in /var/run/openvswitch and on Ubuntu they are created in /usr/var/run/openvswitch I am not sure if the information above will resolve your issues but as I said at the start of the email we are currently working to update our scripts to enable ubuntu support. Regards Sean. -----Original Message----- From: Gabe Black [mailto:gabe.bl...@viavisolutions.com] Sent: Wednesday, August 19, 2015 12:33 AM To: Mooney, Sean K Cc: John Lange; b...@openvswitch.org Subject: DPDK OVS on Ubuntu 15.04 Hi Sean, I have carried on where my colleague John Lange left off. I have installed Ubuntu 15.04 which has kilo support but instead have used the devstack from https://github.com/openstack-dev/devstack to install everything. For now, I am trying to get a single host set up to try and keep it as simple as possible. I have used the sample configuration file you suggested for an all in one node provided here https://github.com/stackforge/networking-ovs-dpdk/blob/master/doc/source/_downloads/local.conf_example Changes that I have made in order to get things to work on Ubuntu, are the following: ==================================================== networking-ovs-dpdk/devstack/ovs-dpdk/ovs-dpdk-init:454 + qemu_group=`id -ng $qemu_group` Also lines 463, and 471 I changed the command running in the screen from running "sudo su -g $qemu_group ..." to "sg $qemu_group ..." ==================================================== The reason for these changes is that on Ubuntu the "-g" option does not exist on the su command, so /etc/init.d/ovs-dpdk fails to start. Also the name "libvirt-qemu" is not the group name, but rather the user. It seems that with those changes devstack.sh is be able to complete. -------------------------------- There were a few messages that made me wonder if all was well; for example the ovs-vswitchd.log file had the following: ... 2015-08-18T22:20:28Z|00011|dpif_netlink|ERR|Generic Netlink family 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably not loaded. ... Also, creating the vhost user port resulted in the following: (note, the ovs-vsctl add-port... command to create the vhost-user did not complain) ... VHOST_CONFIG: socket created, fd:52 VHOST_CONFIG: bind to /usr/var/run/openvswitch/vhost-user-1 2015-08-18T22:33:01Z|00077|dpdk|INFO|Socket /usr/var/run/openvswitch/vhost-user-1 created for vhost-user port vhost-user-1 2015-08-18T22:33:01Z|00078|dpif_netdev|ERR|Cannot create pmd threads due to out of unpinned cores on numa node 2015-08-18T22:33:01Z|00079|bridge|INFO|bridge br-p6p1: added interface vhost-user-1 on port 3 ... I never attempted to pin any cores, but it made me wonder if something might be incomplete... (I'm running on a dual quad core (with hyperthreading)). ===================================================== Anyway, I added the following sections to my VM's libvirt qemu xml (/etc/libvirt/qemu/ubuntutrusty.xml) file to try and have it take the vhost-user interface, as well as ensure the VM is backed by hugetable memory. These are the sections I added: Hugetable-backed memory: <memoryBacking> <hugepages/> </memoryBacking> Vhost-user interface (I tried using the <interface type="vhostuser">... but virsh edit would complain about it not validating against domain.rng... I thought it would be supported... running 1.2.12 ) <qemu:commandline> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=char1,path=/usr/var/run/openvswitch/vhost-user-1'/> <qemu:arg value='-netdev'/> <qemu:arg value='type=vhost-user,id=mynet1,chardev=char1,vhostforce'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-net-pci,mac=54:53:00:6a:b3:00,netdev=mynet1'/> </qemu:commandline> Ok, so this finally brings me to my query for assistance: When I try and launch the VM (via virt-manager), it complains about two things: 1) "process exited while connecting to monitor: /usr/bin/kvm-spice: line 36: /tmp/qemu.orig: Permission denied" - I tried chmod 777 that file just to allow anyone to write/access that file, and it is getting the command written to it, but that error message still shows... Don't know if that is important.. 2) qemu-system-x86_64: -chardev socket,id=char1,path=/usr/var/run/openvswitch/vhost-user-1: Failed to connect to socket: Permission Denied. I've tried messing with group/user config variables in /etc/libvirt/qemu.conf and /etc/libvirt/libvirtd.conf but most of the time I then get devstack's n-cpu unable to connect to libvirtd (similar permission error). I feel I am close, and would appreciate any advice you might have. Very best, Gabriel Black
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
