Hugo: 1. Your instructions here are for CentOS 6.4 environment.
However, ASF 4.2 Cloudstack is currently only supported on CentOS 6.3, ** NOT CentOS 6.4 ** Since ASF 4.2 Cloudstack has product features requiring openvswitch support , That is why we need to have openvswitch (1.7.1 probably) running on CentOS 6.3 for ASF 4.2 Cloudstack. ** NOT CENTOS 6.4 ** Also, ASF 4.2 Cloudstack has not been tested on CentOS 6.4 yet. There are no ASF 4.2 cloudstack builds available on CentOS 6.4 We need ASF 4.2 Cloudstack openvswitch running and tested on CENTOS 6.3. Therefore I have been installing libvirt-0.10.1 and openvswitch-1.7.1 on CentOS 6.3 in CentOS 6.3 Will your configurations work on CentOS 6.3 ? 2. Your instruction in installing cloudstack-agent: # yum install cloudstack-agent What version of cloudstack-agent are you installing? Please show content of your /etc/repos.d/<repofile> ? There is no cloudstack-agent on CentOS 6.4 yet. 2. What is the section on ============== centos64-openvswitch.patch ========== about? 3. In CentOS 6.3, only libvirt-0.9.10 is available. Cloudstack-agent installation installs libvirt-0.9.10 Therefore I had to install and overlay libvirt-0.10.1 over libvirt-0.9.10 4. Your response to my previous email about your openvswitch example To my question of commands for openswitch configuration : > ovs-vsctl add-br br0 > ovs-vsctl add-port br0 eth1 The redhat network configuration should take care of creating all the bridges and linking the ports. But yes the configuration above should have the same effect as these two commands. Do you mean there is NO NEED to execute these two commands at all as long as the files are configured, Because these commands are listed in ACS installation guide for openvswitch configuration, and upon executing the Second command ovs-vsctl add-port br0 eth1 the host lost network connection ? Thanks -----Original Message----- From: Hugo Trippaers [mailto:htrippa...@schubergphilis.com] Sent: Thursday, April 25, 2013 6:19 AM To: Angeline Shen; '<dev@cloudstack.apache.org>'; Sheng Yang; Edison Su Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS support in KVM Heya, Comments inline. I ran through the entire procedure I use to test the setup and documented it as detailed as possible. Hope this helps: # # CentOS KVM installation # Base installation: CentOS 6.2 Basic Server Upgraded to CentOS 6.4: # yum update Install Virtualization tools # yum groupinstall Virtualization "Virtualization Client" "Virtualization Platform" "Virtualization Tools" Verify Installed: hypervkvpd.x86_64 0:0-0.9.el6 libguestfs.x86_64 1:1.16.34-2.el6 libvirt.x86_64 0:0.10.2-18.el6_4.3 libvirt-client.x86_64 0:0.10.2-18.el6_4.3 python-virtinst.noarch 0:0.600.0-15.el6 qemu-kvm.x86_64 2:0.12.1.2-2.355.0.1.el6.centos.2 virt-manager.x86_64 0:0.9.0-18.el6 virt-top.x86_64 0:1.0.4-3.15.el6 virt-viewer.x86_64 0:0.5.2-18.el6_4.2 Modify libvirt configuration: Edit /etc/libvirt/libvirtd.conf listen_tls = 0 listen_tcp = 1 tcp_port = 16059 auth_tcp = "none" mdns_adv = 0 Edit /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--listen" Start libvirtd # /etc/init.d/libvirtd start # /etc/init.d/libvirtd status Verify installation: # virsh capabilities - Should list two guest tags with os_type hvm Build and install openvswitch: Install build requirements: # yum install rpmdevtools openssl-devel kernel-devel gcc redhat-rpm-config Build packages: # mkdir -p ~/rpmbuild/SOURCES # curl -O http://openvswitch.org/releases/openvswitch-1.9.0.tar.gz # cp openvswitch-1.9.0.tar.gz ~/rpmbuild/SOURCES # cp centos64-openvswitch.patch ~/rpmbuild/SOURCES # tar -xzf openvswitch-1.9.0.tar.gz # cd openvswitch-1.9.0 # patch -p1 < ~/rpmbuild/SOURCES/centos64-openvswitch.patch # rpmbuild -bb rhel/openvswitch.spec # rpmbuild -bb -D "kversion `uname -r`" rhel/openvswitch-kmod-rhel6.spec Install openvswitch: # yum install ~/rpmbuild/RPMS/x86_64/kmod-openvswitch-1.9.0-1.el6.x86_64.rpm ~/rpmbuild/RPMS/x86_64/openvswitch-1.9.0-1.x86_64.rpm # echo 'blacklist bridge' >> /etc/modprobe.d/blacklist.conf # reboot Verify installation: # lsmod |grep openvswitch # ovs-vsctl -V Network design: cloudbr0 (Management, Storage) ip: 172.16.10.10/24 gateway: 172.16.10.1 eth0 (physical port, no vlans) cloudbr1 (Guest, Public) eth1 (physical port, vlan trunk) ip: none Configure network interfaces: /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none IPV6INIT=no NM_CONTROLLED=no ONBOOT=yes TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=cloudbr0 /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1 BOOTPROTO=none IPV6INIT=no NM_CONTROLLED=no ONBOOT=yes TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=cloudbr1 /etc/sysconfig/network-scripts/ifcfg-cloudbr0 DEVICE=cloudbr0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static IPADDR=172.16.10.10 GATEWAY=172.16.10.1 NETMASK=255.255.255.0 HOTPLUG=no /etc/sysconfig/network-scripts/ifcfg-cloudbr1 DEVICE=cloudbr1 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=none HOTPLUG=no /etc/sysconfig/network NETWORKING=yes HOSTNAME=testkvm1 GATEWAY=172.10.10.1 Install cloudstack-agent: # yum install cloudstack-agent Edit /etc/cloudstack/agent/agent.properties network.bridge.type=openvswitch libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver Now add the host to cloudstack. ================ centos64-openvswitch.patch ===================== diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h --- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h 2013-02-26 21:25:37.000000000 +0100 +++ openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h +++ 2013-04-25 10:45:09.942027933 +0200 @@ -4,16 +4,4 @@ #include <linux/version.h> #include_next <linux/etherdevice.h> -#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,36) -static inline void eth_hw_addr_random(struct net_device *dev) -{ - random_ether_addr(dev->dev_addr); -} -#elif LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0) -static inline void eth_hw_addr_random(struct net_device *dev) -{ - dev_hw_addr_random(dev, dev->dev_addr); -} -#endif - #endif diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h --- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h 2013-02-26 21:25:37.000000000 +0100 +++ openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h 2013-04-25 10:44:33.270023182 +0200 @@ -55,38 +55,4 @@ #define VLAN_TAG_PRESENT VLAN_CFI_MASK #endif -#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) -static inline void vlan_set_encap_proto(struct sk_buff *skb, struct vlan_hdr *vhdr) -{ - __be16 proto; - unsigned char *rawp; - - /* - * Was a VLAN packet, grab the encapsulated protocol, which the layer - * three protocols care about. - */ - - proto = vhdr->h_vlan_encapsulated_proto; - if (ntohs(proto) >= 1536) { - skb->protocol = proto; - return; - } - - rawp = skb->data; - if (*(unsigned short *) rawp == 0xFFFF) - /* - * This is a magic hack to spot IPX packets. Older Novell - * breaks the protocol design and runs IPX over 802.3 without - * an 802.2 LLC layer. We look for FFFF which isn't a used - * 802.2 SSAP/DSAP. This won't work for fault tolerant netware - * but does for the rest. - */ - skb->protocol = htons(ETH_P_802_3); - else - /* - * Real 802.2 LLC - */ - skb->protocol = htons(ETH_P_802_2); -} -#endif #endif /* linux/if_vlan.h wrapper */ diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h --- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h 2013-02-26 21:25:37.000000000 +0100 +++ openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h 2013-04-25 10:43:21.167021697 +0200 @@ -245,10 +245,4 @@ } #endif -#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,40) -static inline void skb_reset_mac_len(struct sk_buff *skb) -{ - skb->mac_len = skb->network_header - skb->mac_header; -} -#endif #endif diff -ru openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec --- openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec 2013-02-26 21:25:52.000000000 +0100 +++ openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec 2013-04-25 +++ 10:40:41.039023915 +0200 @@ -22,6 +22,7 @@ Source0: %{oname}-%{version}.tar.gz BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX) BuildRequires: %kernel_module_package_buildreqs +Patch0: centos64-openvswitch.patch # Without this we get an empty openvswitch-debuginfo package (whose name # conflicts with the openvswitch-debuginfo package for OVS userspace). @@ -43,6 +44,7 @@ %prep %setup -n %{oname}-%{version} +%patch0 -p1 %build for flavor in %flavors_to_build; do ================ centos64-openvswitch.patch ===================== > -----Original Message----- > From: Angeline Shen [mailto:angeline.s...@citrix.com] > Sent: Thursday, April 25, 2013 5:31 AM > To: '<dev@cloudstack.apache.org>'; Hugo Trippaers; Sheng Yang; Edison > Su > Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS > support in KVM > > Hugo: > > For your configuration example : > > > # The physical interface: eth1 > DEVICE=eth1 > BOOTPROTO=none > HWADDR=BC:30:5B:D4:16:3C > IPV6INIT=no > #MTU=1500 > NM_CONTROLLED=no > ONBOOT=yes > TYPE=OVSPort > DEVICETYPE=ovs > OVS_BRIDGE=br0 > > > # The bridge interface: br0 > DEVICE=br0 > ONBOOT=yes > DEVICETYPE=ovs > TYPE=OVSBridge > BOOTPROTO=static > IPADDR= 10.223.58.195 > GATEWAY = 10.223.58.193 > NETMASK=255.255.255.192 > HOTPLUG=no > > QUESTION: will commands to configure openvswitch network interfaces be > as follows: > > ovs-vsctl add-br br0 > ovs-vsctl add-port br0 eth1 > The redhat network configuration should take care of creating all the bridges and linking the ports. But yes the configuration above should have the same effect as these two commands. > > Thanks > > -----Original Message----- > From: Angeline Shen > Sent: Wednesday, April 24, 2013 1:13 PM > To: <dev@cloudstack.apache.org>; Hugo Trippaers; Sheng Yang; Edison Su > Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS > support in KVM > > Hugo: > > Please review my updates to > https://issues.apache.org/jira/browse/CLOUDSTACK-101 . > > The latest experiment I conducted per your instructions is as follows: > > ovs Host 10.223.58.195 gateway 10.223.58.193 other > host on > same gateway but not in ovs setup 10.223.58.194 > > On ovs host 10.223.58.195 , After I changed /etc/sysconfig/network- > scripts/ifcfg-eth1 and /etc/sysconfig/network-scripts/ifcfg-cloudbr > to your instructions, service network restart , > > the ovs host can now ONLY ping its own gateway 10.223.58.193 and any > host in same subnet, but CANNOT reach anything beyond the gateway > 10.223.58.193. > > 1. /etc/sysconfig/network-scripts/ifcfg-eth1: > > DEVICE=eth1 > BOOTPROTO=none > HWADDR=BC:30:5B:D4:16:3C > IPV6INIT=no > #MTU=1500 > NM_CONTROLLED=no > ONBOOT=yes > #TYPE=Ethernet > TYPE=OVSPort > DEVICETYPE=ovs > OVS_BRIDGE=cloudbr > #UUID="0142ba15-de11-499c-a1ba-82837b30f890" > #IPADDR=10.223.58.195 > #NETMASK=255.255.255.192 > #GATEWAY=10.223.58.193 > #DNS1=10.223.110.254 > #USERCTL=yes > #IPV4_FAILURE_FATAL=yes > #DEFROUTE=yes > #NAME="System eth1" > > 2. /etc/sysconfig/network-scripts/ifcfg-cloudbr: > > DEVICE=cloudbr > ONBOOT=yes > DEVICETYPE=ovs > TYPE=OVSBridge > BOOTPROTO=static > IPADDR=10.223.58.195 > GATEWAY=10.23.58.193 > NETMASK=255.255.255.0 > HOTPLUG=no > > 3. On host 10.223.58.195, service network restart > > > 4. on host 10.223.58.195: > > [root@Rack3Host16 ~]# ip route > 10.223.58.0/24 dev cloudbr proto kernel scope link src 10.223.58.195 > 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1 > 169.254.0.0/16 dev eth1 scope link metric 1002 > 169.254.0.0/16 dev cloudbr scope link metric 1015 > > 5. It can reach its gateway & other hosts on same gateway: > > [root@Rack3Host16 ~]# ping 10.223.58.193 PING 10.223.58.193 > (10.223.58.193) 56(84) bytes of data. > 64 bytes from 10.223.58.193: icmp_seq=1 ttl=64 time=7.43 ms > 64 bytes from 10.223.58.193: icmp_seq=2 ttl=64 time=2.82 ms > 64 bytes from 10.223.58.193: icmp_seq=3 ttl=64 time=1.06 ms ^C > --- 10.223.58.193 ping statistics --- > 3 packets transmitted, 3 received, 0% packet loss, time 2364ms rtt > min/avg/max/mdev = 1.061/3.771/7.430/2.685 ms > [root@Rack3Host16 ~]# ping 10.223.58.194 PING 10.223.58.194 > (10.223.58.194) 56(84) bytes of data. > 64 bytes from 10.223.58.194: icmp_seq=1 ttl=64 time=0.399 ms > 64 bytes from 10.223.58.194: icmp_seq=2 ttl=64 time=0.181 ms ^C > --- 10.223.58.194 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 1446ms rtt > min/avg/max/mdev = 0.181/0.290/0.399/0.109 ms > > [root@Rack3Host16 ~]# traceroute 10.223.58.194 traceroute to > 10.223.58.194 (10.223.58.194), 30 hops max, 60 byte packets > 1 10.223.58.194 (10.223.58.194) 0.922 ms 0.851 ms 0.830 ms > > [root@Rack3Host16 ~]# traceroute 10.223.58.193 traceroute to > 10.223.58.193 (10.223.58.193), 30 hops max, 60 byte packets > 1 10.223.58.193 (10.223.58.193) 7.017 ms 12.506 ms 7.675 ms > > 6. host CANNOT reach anything outside of its subnet: > > [root@Rack3Host16 ~]# ping 10.223.195.114 > connect: Network is unreachable > > [root@Rack3Host16 ~]# ping www.google.com > ping: unknown host www.google.com > > [root@Rack3Host16 ~]# ping 8.8.8.8 > connect: Network is unreachable > > [root@Rack3Host16 ~]# traceroute 8.8.8.8 traceroute to 8.8.8.8 > (8.8.8.8), 30 hops max, 60 byte packets > connect: Network is unreachable > > 7. [root@Rack3Host16 ~]# ifconfig > cloudbr Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C > inet addr:10.223.58.195 Bcast:10.223.58.255 Mask:255.255.255.0 > inet6 addr: fc00:2::be30:5bff:fed4:163c/64 Scope:Global > inet6 addr: fe80::68f7:52ff:fe03:a744/64 Scope:Link > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:46945 errors:0 dropped:0 overruns:0 frame:0 > TX packets:549 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:39235509 (37.4 MiB) TX bytes:102773 (100.3 KiB) > > cloud0 Link encap:Ethernet HWaddr AA:81:04:5F:5E:4A > inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 > inet6 addr: fe80::a881:4ff:fe5f:5e4a/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:23 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:4201 (4.1 KiB) > > eth1 Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C > inet6 addr: fe80::be30:5bff:fed4:163c/64 Scope:Link > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:48123 errors:0 dropped:0 overruns:0 frame:0 > TX packets:532 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:39518352 (37.6 MiB) TX bytes:101960 (99.5 KiB) > Interrupt:16 Memory:da000000-da012800 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:340 errors:0 dropped:0 overruns:0 frame:0 > TX packets:340 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:46264 (45.1 KiB) TX bytes:46264 (45.1 KiB) > > 8. [root@Rack3Host16 ~]# brctl show > bridge name bridge id STP enabled interfaces > cloud0 /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > /sys/class/net/cloud0/bridge: No such file or directory > 0000.aa81045f5e4a no cloudbr /sys/class/net/cloudbr/bridge: No such > file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > /sys/class/net/cloudbr/bridge: No such file or directory > 0000.bc305bd4163c no > > > -----Original Message----- > From: Sheng Yang [mailto:sh...@yasker.org] > Sent: Wednesday, April 24, 2013 12:59 PM > To: Hugo Trippaers > Cc: <dev@cloudstack.apache.org> > Subject: Re: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS > support in KVM > > Thank you Hugo! > > --Sheng > > > On Wed, Apr 24, 2013 at 12:16 PM, Hugo Trippaers < > htrippa...@schubergphilis.com> wrote: > > > Hey Sheng, > > > > Sure thing, I'm trying to help Angeline already, but it's quite a > > complex setup. > > > > I'll try to write down some detailed instructions. > > > > Cheers, > > > > Hugo > > > > Sent from my iPhone > > > > On 24 apr. 2013, at 19:56, "Sheng Yang" <sh...@yasker.org> wrote: > > > > Hi Hugo, > > > > Could you help with KVM OVS setup? I cannot find much information > > on the our wiki about OVS on KVM. > > > > --Sheng > > > > On Fri, Apr 19, 2013 at 8:14 PM, Angeline Shen > <angeline.s...@citrix.com>wrote: > > > >> ASF 4.1 cloudstack with openvswitch support for RHEL 6.3 KVM test > >> configuration setup: > >> > >> After completing > >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on- > cent > >> os-6-3-minimal/ to setup RHEL 6.3 openvswitch, encountered > >> following > >> problems: > >> > >> 1. In this scenario, after service openvswitch start, host is still > >> up running but completely lost connection to network. > >> > >> [root@Rack3Host16 /]# service openvswitch start Inserting brcompat > >> module [ OK ] Starting ovsdb-server [ OK ] Configuring Open vSwitch > >> system IDs [ OK ] Starting ovs-vswitchd [ OK ] Starting > >> ovs-brcompatd [ OK ] iptables already has a rule for gre, not explicitly > >> enabling. > >> [root@Rack3Host16 /]# > >> > >> > >> 2. content of various files: > >> > >> /etc/sysconfig/network-scripts/ifcfg-ovsbr1 : > >> DEVICE=ovsbr1 > >> ONBOOT=yes > >> DEVICETYPE=ovs > >> TYPE=OVSBridge > >> BOOTPROTO=static > >> IPADDR=10.223.58.195 > >> GATEWAY=10.223.58.193 > >> NETMASK=255.255.255.0 > >> HOTPLUG=no > >> > >> > >> QUESTION: Is > >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on- > centos-6-3-minimal/ > >> Valid configuration instructions for openvswitch RHEL 6.3 > >> or are there other updated correct configuration steps? > >> > >> Thanks > >> > >> > >