Hi Thiago, It would be interesting to see the throughput results using something like iperf between (other vms on different Hypervisors, vms on the same hypervisor and between a vm and the router namespace). This should help in pin pointing the issue.
Aaron On Tue, Oct 22, 2013 at 9:17 PM, Martinx - ジェームズ <thiagocmarti...@gmail.com>wrote: > Sure, always... For example, from within a tenant instance: > > ------ > ubuntu@instance-1:~$ ip route > default via 192.168.210.1 dev eth0 metric 100 > 192.168.210.0/24 dev eth0 proto kernel scope link src 192.168.210.2 > > # 192.168.210.1 resides within the Tenant Namespace at the Network Node, > ping ok: > ubuntu@instance-1-1:~$ ping -c 1 192.168.210.1 > > > PING 192.168.210.1 (192.168.210.1) 56(84) bytes of data. > 64 bytes from 192.168.210.1: icmp_req=1 ttl=64 time=0.898 ms > > --- 192.168.210.1 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.898/0.898/0.898/0.000 ms > > # Internet... > ubuntu@instance-1:~$ ping -c 1 8.8.8.8 > > > PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. > 64 bytes from 8.8.8.8: icmp_req=1 ttl=48 time=133 ms > > --- 8.8.8.8 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 133.232/133.232/133.232/0.000 ms > > # Very slow speed: > ubuntu@instance-1:~$ sudo aptitude update > Get: 1 http://nova.clouds.archive.ubuntu.com precise Release.gpg [198 B] > Get: 2 http://security.ubuntu.com precise-security Release.gpg [198 B] > Get: 3 http://nova.clouds.archive.ubuntu.com precise-updates Release.gpg > [198 B] > Get: 4 http://security.ubuntu.com precise-security Release [49.6 kB] > Get: 5 http://nova.clouds.archive.ubuntu.com precise-backports > Release.gpg [198 B] > Hit http://nova.clouds.archive.ubuntu.com precise Release > Get: 6 http://nova.clouds.archive.ubuntu.com precise-updates Release > [49.6 kB] > 81% [6 Release 44.5 kB/49.6 kB 90%] [4 Release 35.4 kB/49.6 kB 71%] > > > *7,521 B/s 2s* > ------ > > 7,521 B/s ??? > > If I run "aptitude update" from within Tenant's Namespace, the "router" > with the IP 192.168.210.1 (tenant's gateway), it goes just fine. Look: > > ------ > root@net-node-1:~# ip netns exec > qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 ip r > default via 172.16.0.1 dev qg-50b615b7-c2 > 172.16.0.0/20 dev qg-50b615b7-c2 proto kernel scope link src > 172.16.0.2 > 192.168.210.0/24 dev qr-a1376f61-05 proto kernel scope link src > 192.168.210.1 > > # Normal speed > root@net-node-1:~# ip netns exec > qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 aptitude update > Hit http://us.archive.ubuntu.com precise Release.gpg > > Hit http://us.archive.ubuntu.com precise-updates Release.gpg > Get: 1 http://us.archive.ubuntu.com precise-backports Release.gpg [198 B] > Get: 2 http://ubuntu-cloud.archive.canonical.com precise-updates/havana > Release.gpg [543 B] > Get: 3 http://security.ubuntu.com precise-security Release.gpg [198 B] > ....sniped.... > ------ > > No idea about what's happening... ;-( > > Also, the Compute Node have normal speed too, where the instance-1 is > running. > > This Havana I'm trying to put into production, follow a well tested > topology that I already have up and running, very similar with this: > https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst > > I'll (double * double) check everything again. I like to test and debug... > :-D > > Best, > Thiago > > > On 23 October 2013 01:52, 止语 <menghuizhan...@gmail.com> wrote: > >> Have you ever try to check the route ? >> #route -n >> >> >> ------------------ Original ------------------ >> *From: * "Martinx - ジェームズ";<thiagocmarti...@gmail.com>; >> *Date: * Wed, Oct 23, 2013 11:34 AM >> *To: * "Geraint Jones"<gera...@koding.com>; ** >> *Cc: * "openstack@lists.openstack.org"<openstack@lists.openstack.org>; ** >> *Subject: * Re: [Openstack] Very slow connectivity from within tenant >> network -GRE >> >> WOW! Nice move! >> >> But, upgrading from `openvswitch 1.10.2-0ubuntu2~cloud0` to `1.11.0-1` >> did not solved my issue. >> >> Tenant Instances still have a very slow Internet connectivity. >> >> Thanks anyway! Nice to see your charts, pretty good improvement! >> >> Regards, >> Thiago >> >> >> On 22 October 2013 22:48, Geraint Jones <gera...@koding.com> wrote: >> >>> I have just had to tweak our grizzly network node the biggest impacts >>> were seem from doing changing root wrap to only use sudo – not the python >>> wrapper (its super slow) and upgrading openvswitch to 1.11 >>> >>> This smoke ping shows the latency to one of our instances from europe >>> before and after the openvswitch upgrade : http://d.pr/i/36v0 >>> >>> And this graph shows the load avg on our network node, the first drop is >>> from disabling root wrap the second is after the OVS upgrade : >>> http://d.pr/i/xhFc >>> >>> I would suggest you do the same, and just make sure all MTU’s are >>> correct. >>> >>> -- >>> Geraint Jones >>> Director of Systems & Infrastructure >>> Koding >>> https://koding.com >>> gera...@koding.com >>> M (NZ) +64 22 123 4626 >>> M (US) +1 415 316 8027 >>> >>> From: Martinx - ジェームズ <thiagocmarti...@gmail.com> >>> Date: Tuesday, 22 October 2013 9:00 am >>> To: Rick Jones <rick.jon...@hp.com> >>> Cc: "openstack@lists.openstack.org" <openstack@lists.openstack.org> >>> Subject: Re: [Openstack] Very slow connectivity from within tenant >>> network - GRE >>> >>> Hi Rick! >>> >>> Back with Grizzly, I faced that problem and I was able to detect it, at >>> the Network Node with tcpdump and fix it by running "ip link set mtu 1454 >>> dev eth0" within the Instance. >>> >>> Not this time... This is another problem... ;-/ >>> >>> >>> On 22 October 2013 13:25, Rick Jones <rick.jon...@hp.com> wrote: >>> >>>> On 10/22/2013 01:32 AM, Martinx - ジェームズ wrote: >>>> >>>>> Stackers, >>>>> >>>>> I'm trying to put my Havana into production and I'm facing a very >>>>> strange problem. >>>>> >>>>> The Internet connectivity from tenant's subnet is very, very slow. It >>>>> is >>>>> useless in fact... I can not even use "apt-get update" from a Instance. >>>>> >>>>> The following command works (apt update from the tenant namespace): >>>>> >>>>> --- >>>>> root@net-node-1:~# ip netns exec qrouter-XXXXXXXXX aptitude update >>>>> --- >>>>> >>>>> But not from the tenant subnet... >>>>> >>>>> I'm following this topology: >>>>> >>>>> http://docs.openstack.org/**trunk/install-guide/install/** >>>>> apt/content/section_use-cases-**tenant-router.html<http://docs.openstack.org/trunk/install-guide/install/apt/content/section_use-cases-tenant-router.html> >>>>> >>>>> Already tried to change MTUs (via DHCP agent)... Nothing had fixed this >>>>> weird issue. >>>>> >>>>> Any thoughts?! >>>>> >>>>> Right now, my "aptitude safe-upgrade" will take 2 days to download >>>>> 60MB... During this network outages, even the SSH session stops >>>>> responding for a few seconds... >>>>> >>>>> Everything else seems to be working as expected, as for example, DHCP, >>>>> Floating IPs, Security Groups... >>>>> >>>>> Sometimes, even the first ssh connection to the Instance Floating IP, >>>>> have a lag. >>>>> >>>> >>>> It is but a guess, but I wonder if, even with changing MTUs (to what >>>> values?) you may still be experiencing a PathMTU+ICMP blackhole problem >>>> accessing nodes on the Internet. Can you access something that is a bit >>>> "closer" but still outside your stack so you have a shot at looking at >>>> netstat statistics on the sender and/or get packet traces on the sender? >>>> >>>> You could still try taking packet traces at the instance or perhaps the >>>> namespace and try to discern packet losses at the receiving side, though it >>>> can be a bit more difficult. >>>> >>>> rick jones >>>> >>>> >>> _______________________________________________ Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : >>> openstack@lists.openstack.org Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >> >> > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
