Guys,
In fact, the "ethtool" workaround is far from perfect, I'm still seeing
lots of connectivity issues within my KVM Host and its Guests, even after
disabling gso/tso/tx on all interfaces, both virtual (within guests) and
physical (at the host).
I really appreciate any help!
Thanks!
Thiago
On 19 October 2014 03:30, Martinx - ジェームズ <thiagocmarti...@gmail.com> wrote:
> Oops!
>
> It seems to be working now! :-D
>
> This that I just said:
>
> I tried to disable tso/gso/tx at another guest, of vlan100 net, didn't
>> worked either.
>>
>
> Isn't true... I created a new guest, side-by-side with "guest-fw-1", on
> vlan100, and after disabling `tso/gso/tx`, its connectivity becomes stable!
>
> I'll try this: git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>
> Nevertheless, I'm bit more confused now, the problem was fixed by
> disabling `tso/gso/tx` but, I did this *only within this new
> "guest-server-1"* that I created within vlan100... Only at it, I disabled
> "tso/gso/tx", so, the problem seems to not be within the "guest-fw-1" as I
> was thinking... Weird...
>
> Tks,
> Thiago
>
>
>> Regards,
>> Thiago
>>
>> On 29 August 2014 11:20, Vlad Yasevich <vyase...@redhat.com> wrote:
>>
>>> [ realized that the bug and reporter were non cc'd, updated cc list]
>>>
>>> On 08/28/2014 02:40 PM, Thiago Martins wrote:
>>> > Public bug reported:
>>> >
>>> > Guys,
>>> >
>>> > Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
>>> network.
>>> > It does not route tagged VLAN packets.
>>> >
>>>
>>> The have a been a bunch of rather recent changes to the kernel to support
>>> guest VLANs correctly. The issues have been around TSO/GSO
>>> implementation
>>> in the kernel.
>>>
>>> Could try disabling TSO/GSO and tx checksums on the vlan devices in the
>>> guest
>>> and see if it solves your problem?
>>>
>>> If it does, could you try the kernel from
>>> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>>> turn the offloads back on and see if the problem is solved?
>>>
>>> Thanks
>>> -vlad
>>>
>>> > That's it, it is impossible to use Trusty acting as a QEmu 2.0
>>> > Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
>>> > tagged network within itself. QEmu 2.X guest does not route traffic
>>> when
>>> > with tagged VLANs!
>>> >
>>> > So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
>>> > "firewalls / routers", and it have an easy to reproduce, connectivity
>>> > problem.
>>> >
>>> > This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
>>> > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>>> >
>>> > I have this very same setup up and running, on about ~100 physical
>>> > servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
>>> them,
>>> > the QEmu Hypervisors dedicated to host "guest acting as routers /
>>> > firewalls", like a "borger gateway" for example, that it does not work
>>> > as expected.
>>> >
>>> > One interesting thing to note is that, this BUG appear only, and only
>>> > at, the QEmu Hypervisors dedicated to host guests that are used as
>>> > `router / firewalls` (as I said above), others QEmu Hypervisors of my
>>> > network does not suffer from this problem.
>>> >
>>> > Another interesting point is that it fails to route tagged VLAN packets
>>> > only when these packets are originated from within the Hypervisor
>>> > itself, I mean, packets from both host and other guests (not the
>>> > router/firewall guest itself), suffer from this connectivity problem.
>>> >
>>> > As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
>>> > "border hypervisor". So, this proves that there is something wrong with
>>> > QEmu.
>>> >
>>> > I already tested it with both `openvswitch-switch` and with `bridge-
>>> > utils`, same bad results. So, don't waste your time trying `bridge-
>>> > utils` (optional steps while reproducing it), you can keep OVS bridges
>>> > from original design.
>>> >
>>> > I think that I'm using the best pratices to build this environment, as
>>> > follows...
>>> >
>>> >
>>> > * Topology *
>>> >
>>> >
>>> > QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
>>> hypervisor"):
>>> >
>>> > 1- Physical machine with 3 NICs;
>>> > 2- Minimal Ubuntu 14.04.1 installed and upgraded;
>>> > 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd
>>> tcpdump".
>>> >
>>> > - eth0 connected to the Internet - VLAN tag 10;
>>> > - eth1 connected to the LAN1 - VLAN tag 100;
>>> > - eth2 connected to the LAN2 - VLAN tag 200;
>>> >
>>> >
>>> > Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
>>> guest acting as a router with iptables/ip6tables):
>>> >
>>> > 1- Virtual Machine with 3 NICs (VirtIO);
>>> > 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
>>> > 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
>>> >
>>> >
>>> > OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
>>> > `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
>>> > (virt-)lab network, including the `qemu-host-1` itself.
>>> >
>>> >
>>> > Steps to reproduce
>>> >
>>> >
>>> > * Preparing the `qemu-host-1` host:
>>> >
>>> > - Configure the /etc/network/interfaces with:
>>> >
>>> > ---
>>> > # The loopback network interface
>>> > auto lo
>>> > iface lo inet loopback
>>> >
>>> > auto eth0
>>> > iface eth0 inet manual
>>> > up ip link set $IFACE up
>>> > down ip link set $IFACE down
>>> >
>>> > auto eth1
>>> > iface eth1 inet manual
>>> > up ip link set dev $IFACE up
>>> > down ip link set dev $IFACE down
>>> >
>>> > auto ovsbr1p1
>>> > iface ovsbr1p1 inet6 auto
>>> >
>>> > iface ovsbr1p1 inet static
>>> > address 192.168.1.10
>>> > netmask 24
>>> > gateway 192.168.1.1
>>> >
>>> > auto eth2
>>> > iface eth2 inet manual
>>> > up ip link set $IFACE up
>>> > down ip link set $IFACE down
>>> > ---
>>> >
>>> >
>>> > - Creating the Hypervisor OVS Bridges:
>>> >
>>> > ovs-vsctl add-br ovsbr0
>>> > ovs-vsctl add-br ovsbr1
>>> > ovs-vsctl add-br ovsbr2
>>> >
>>> >
>>> > - Attaching the bridges to the NICs:
>>> >
>>> > ovs-vsctl add-port ovsbr0 eth0
>>> > ovs-vsctl add-port ovsbr1 eth1
>>> > ovs-vsctl add-port ovsbr2 eth2
>>> >
>>> >
>>> > - Creating the OVS internal tagged interface (best practice?), so the
>>> QEmu Hypervisor itself can have its own IP (v4 and v6):
>>> >
>>> > ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1
>>> type=internal
>>> > ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
>>> >
>>> >
>>> > NOTE:
>>> >
>>> > * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
>>> > with SLAAC, so, it remain fixed across host reboots.
>>> >
>>> >
>>> > - Making Libvirt aware of OVS Bridges:
>>> >
>>> > Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
>>> > and ovsbr2.xml):
>>> >
>>> > --- ovsbr0.xml contents:
>>> > <network>
>>> > <name>ovsbr0</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr0'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> > --- ovsbr1.xml contents:
>>> > <network>
>>> > <name>ovsbr1</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr1'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> > --- ovsbr2.xml contents:
>>> > <network>
>>> > <name>ovsbr2</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr2'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> >
>>> > Run:
>>> >
>>> > virsh net-define ovsbr0.xml
>>> > virsh net-define ovsbr1.xml
>>> > virsh net-define ovsbr2.xml
>>> >
>>> > virsh net-autostart ovsbr0
>>> > virsh net-autostart ovsbr1
>>> > virsh net-autostart ovsbr2
>>> >
>>> > virsh net-start ovsbr0
>>> > virsh net-start ovsbr1
>>> > virsh net-start ovsbr2
>>> >
>>> >
>>> > - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
>>> Virtual Machine):
>>> >
>>> > 1- VM Configuration file (network-only / cutted):
>>> >
>>> > ---
>>> > <interface type='network'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source network='ovsbr0'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source network='ovsbr1'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source network='ovsbr2'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
>>> > /etc/network/interfaces file like this:
>>> >
>>> > ---
>>> > auto vlan10
>>> > iface vlan10 inet static
>>> > vlan_raw_device eth0
>>> > address 200.2.1.106
>>> > netmask 29
>>> > gateway 200.2.1.105
>>> > dns-nameserver 8.8.8.8
>>> >
>>> > auto vlan100
>>> > iface vlan100 inet6 static
>>> > vlan_raw_device eth1
>>> > address 2001:129X:2XX:810X::2
>>> > netmask 64
>>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>>> >
>>> > iface vlan100 inet static
>>> > vlan_raw_device eth1
>>> > address 192.168.4.1
>>> > netmask 24
>>> >
>>> > auto vlan200
>>> > iface vlan200 inet6 static
>>> > vlan_raw_device eth2
>>> > address 2001:1291:2de:10::1
>>> > netmask 64
>>> >
>>> > iface vlan200 inet static
>>> > vlan_raw_device eth2
>>> > address 172.16.0.1
>>> > netmask 24
>>> > ---
>>> >
>>> > 3- Enable radvd for your LANs:
>>> >
>>> > ---
>>> > # SERVERS
>>> > interface vlan100 {
>>> > AdvSendAdvert on;
>>> > MinRtrAdvInterval 3;
>>> > MaxRtrAdvInterval 10;
>>> > AdvLinkMTU 1500;
>>> > AdvDefaultPreference high;
>>> > prefix 2001:1291:200:850a::/64 {
>>> > DeprecatePrefix on;
>>> > AdvOnLink on;
>>> > AdvAutonomous on;
>>> > AdvRouterAddr on;
>>> > };
>>> > route ::/0 {
>>> > RemoveRoute on;
>>> > };
>>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>>> > DNSSL domain.com.br { };
>>> > };
>>> > # DESKTOPS
>>> > interface vlan200 {
>>> > AdvSendAdvert on;
>>> > MinRtrAdvInterval 3;
>>> > MaxRtrAdvInterval 10;
>>> > AdvLinkMTU 1500;
>>> > AdvDefaultPreference high;
>>> > prefix 2001:1291:2de:10::/64 {
>>> > DeprecatePrefix on;
>>> > AdvOnLink on;
>>> > AdvAutonomous on;
>>> > AdvRouterAddr on;
>>> > };
>>> > route ::/0 {
>>> > RemoveRoute on;
>>> > };
>>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>>> > DNSSL igcorp.com.br { };
>>> > };
>>> > ---
>>> >
>>> > 4- HIT TUE BUG!
>>> >
>>> > Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
>>> will
>>> > not work! Ping works... TCP connections doesn't.
>>> >
>>> > The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
>>> QEmu
>>> > 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com
>>> `.
>>> >
>>> > Details:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ip r
>>> > default via 192.168.4.1 dev ovsbr1p1
>>> > 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
>>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>>> 192.168.122.1
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86397sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 27sec
>>> >
>>> >
>>> > # ping6 okay...
>>> > root@qemu-host-1:~# ping6 google.com -c1
>>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>>> >
>>> > --- google.com ping statistics ---
>>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>>> >
>>> > # traceroute6 okay...
>>> > root@qemu-host-1:~# traceroute6 google.com
>>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>>> 0.223 ms
>>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>>> 20.738 ms 20.902 ms
>>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>>> 141.706 ms 21.058 ms
>>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>>> ms 20.837 ms 20.903 ms
>>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>>> 24.464 ms 24.649 ms
>>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b)
>>> 24.734 ms 24.525 ms 25.273 ms
>>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms
>>> 36.335 ms
>>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>>> 36.375 ms
>>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms
>>> 31.512 ms
>>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>>> ms 30.769 ms
>>> >
>>> > # NOTE: the second hope is the "guest-fw-1".
>>> >
>>> > # "apt-get update", not okay! *BUG*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>>> [Connecting to sec
>>> >
>>> > # it remains "Waiting for headers" forever...
>>> >
>>> > # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
>>> > shows:
>>> >
>>> > http://pastebin.com/2BUiNEfQ
>>> >
>>> > ---
>>> >
>>> >
>>> > (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
>>> it!)
>>> >
>>> > Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
>>> > `bridge-utils` instead...
>>> >
>>> > * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
>>> > instead of openvswitch-switch.
>>> >
>>> >
>>> > ------------------------
>>> >
>>> > 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
>>> >
>>> > - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>>> >
>>> > ---
>>> > auto br0
>>> > iface br0 inet manual
>>> > bridge_ports eth0
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> >
>>> > auto br1
>>> > iface br1 inet manual
>>> > bridge_ports eth1
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> >
>>> > auto vlan100
>>> > iface vlan100 inet6 auto
>>> > vlan_raw_device br1
>>> >
>>> > iface vlan100 inet static
>>> > vlan_raw_device br1
>>> > address 192.168.1.10
>>> > netmask 24
>>> > gateway 192.168.1.1
>>> >
>>> > auto br2
>>> > iface br2 inet manual
>>> > bridge_ports eth2
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> > ---
>>> >
>>> >
>>> > 2- New VM Configuration file (network-only section / cutted), adjusted
>>> > to make use bridges from `bridge-utils` package:
>>> >
>>> > ---
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source bridge='br0'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source bridge='br1'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source bridge='br2'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > * Start `guest-fw-1` as-is:
>>> >
>>> > virsh start guest-fw-1
>>> >
>>> >
>>> > New try:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ip r
>>> > default via 192.168.4.1 dev vlan100
>>> > 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
>>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>>> 192.168.122.1
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep vlan100
>>> > 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires
>>> 86397sec
>>> > fe80::/64 dev vlan100 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024
>>> expires 27sec
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # ping6 okay...
>>> > root@qemu-host-1:~# ping6 google.com -c1
>>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>>> >
>>> > --- google.com ping statistics ---
>>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>>> >
>>> > # traceroute6 okay...
>>> > root@qemu-host-1:~# traceroute6 google.com
>>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>>> 0.223 ms
>>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>>> 20.738 ms 20.902 ms
>>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>>> 141.706 ms 21.058 ms
>>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>>> ms 20.837 ms 20.903 ms
>>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>>> 24.464 ms 24.649 ms
>>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b)
>>> 24.734 ms 24.525 ms 25.273 ms
>>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms
>>> 36.335 ms
>>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>>> 36.375 ms
>>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms
>>> 31.512 ms
>>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>>> ms 30.769 ms
>>> >
>>> >
>>> > # BUG effect! "apt-get update", not okay!
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>>> [Connecting to sec
>>> >
>>> > # it remains "Waiting for headers" forever...
>>> >
>>> > - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
>>> > falling back to OpenvSwitch as we started.
>>> >
>>> >
>>> > ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
>>> OpenvSwitch.
>>> >
>>> >
>>> > -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>>> >
>>> > ---
>>> > name = "guest-fw-1"
>>> >
>>> > uuid = "17e031c7-1264-4979-8f06-c5e016469474"
>>> >
>>> > bootloader = "pygrub"
>>> >
>>> > memory = 2048
>>> >
>>> > vcpus = 2
>>> >
>>> > vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
>>> > 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
>>> >
>>> > disk = [
>>> 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
>>> > ---
>>> >
>>> > Details - Working as expected when with Xen!! Look:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ping6 -c 1 google.com
>>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>>> >
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # *BUG dissapeared!*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > Ign http://us.archive.ubuntu.com trusty InRelease
>>> > Ign http://security.ubuntu.com trusty-security InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>>> > ........................
>>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>>> > Reading package lists... Done
>>> > ---
>>> >
>>> >
>>> > Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
>>> expected! You guys can see that the `guest-fw-1` is working on top of Xen,
>>> as-is, I mean, the changes happened only at the Hypervisor itself, problem
>>> solved (not for QEmu)!
>>> >
>>> > But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
>>> the
>>> > host `qemu-host-1` cannot browse the web again (`apt-get update` will
>>> > not work if its gateway is a QEmu guest).
>>> >
>>> >
>>> > ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
>>> bridges".
>>> >
>>> > The presented workaround have one big downside, while it allows us to
>>> > keep using QEmu (and KSM), it requires a complete reconfiguration of
>>> the
>>> > `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
>>> > a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
>>> > overhead for your hypervisor as a whole, I'm not sure), plus a lot of
>>> > extra work... If you need to add a new VLAN to your `guest-fw-1`,
>>> you'll
>>> > need to reboot it, to add a new VirtIO NIC (this isn't the best way to
>>> > build hypervisors - not the best practice), this is just a real
>>> > workaround that allows you to keep using QEmu (and benefits from KSM,
>>> > Libvirt and etc)...
>>> >
>>> > While, when replacing QEmu by Xen, you don't need to change a single
>>> > line within the guest itself...
>>> >
>>> > So, this network problem lies within the QEmu Virtual Machine!
>>> >
>>> > Doing this workaround:
>>> >
>>> > 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
>>> >
>>> > ovs-vsctl add-br vlan10 ovsbr0 10
>>> > ovs-vsctl add-br vlan100 ovsbr1 100
>>> > ovs-vsctl add-br vlan200 ovsbr2 100
>>> >
>>> > 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
>>> >
>>> > ---
>>> > <interface type='network'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source network='vlan10'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source network='vlan100'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source network='vlan200'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>>> >
>>> > ---
>>> > auto eth0
>>> > iface eth0 inet static
>>> > # vlan_raw_device eth0
>>> > address 200.2.1.106
>>> > netmask 29
>>> > gateway 200.2.1.105
>>> > dns-nameserver 8.8.8.8
>>> >
>>> > auto eth1
>>> > iface eth1 inet6 static
>>> > # vlan_raw_device eth1
>>> > address 2001:129X:2XX:810X::2
>>> > netmask 64
>>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>>> >
>>> > iface eth1 inet static
>>> > # vlan_raw_device eth1
>>> > address 192.168.4.1
>>> > netmask 24
>>> >
>>> > auto eth2
>>> > iface eth2 inet6 static
>>> > # vlan_raw_device eth2
>>> > address 2001:1291:2de:10::1
>>> > netmask 64
>>> >
>>> > iface eth2 inet static
>>> > # vlan_raw_device eth2
>>> > address 172.16.0.1
>>> > netmask 24
>>> > ---
>>> >
>>> > 4- Details: Working as expected when with QEmu but, without tagging the
>>> > VLAN within the `guest-fw-1` itself.
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ping6 -c 1 google.com
>>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>>> >
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256.
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # *BUG dissapeared!*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > Ign http://us.archive.ubuntu.com trusty InRelease
>>> > Ign http://security.ubuntu.com trusty-security InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>>> > ........................
>>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>>> > Reading package lists... Done
>>> > ---
>>> >
>>> > Conclusion:
>>> >
>>> > A QEmu guest router does not route tagged VLAN packages that are
>>> > originated at its host, neighter from others guests hosted at the same
>>> > hypervisor. Making it impossible to create a virtual network within a
>>> > hypervisor.
>>> >
>>> > Best Regards,
>>> > Thiago Martins
>>> >
>>> > ** Affects: qemu
>>> > Importance: Undecided
>>> > Status: New
>>> >
>>>
>>>
>>>
>>
>
