No prob. Thanks for replying. On May 31, 2017 10:11 AM, "Gustavo Randich" <gustavo.rand...@gmail.com> wrote:
> Hi Kevin, I confirm that applying the patch the problem is fixed. > > Sorry for the inconvenience. > > > On Tue, May 30, 2017 at 9:36 PM, Kevin Benton <ke...@benton.pub> wrote: > >> Do you have that patch already in your environment? If not, can you >> confirm it fixes the issue? >> >> On Tue, May 30, 2017 at 9:49 AM, Gustavo Randich < >> gustavo.rand...@gmail.com> wrote: >> >>> While dumping OVS flows as you suggested, we finally found the cause of >>> the problem: our br-ex OVS bridge lacked the secure fail mode configuration. >>> >>> May be the issue is related to this: https://bugs.launchpad.net/neu >>> tron/+bug/1607787 >>> >>> Thank you >>> >>> >>> On Fri, May 26, 2017 at 6:03 AM, Kevin Benton <ke...@benton.pub> wrote: >>> >>>> Sorry about the long delay. >>>> >>>> Can you dump the OVS flows before and after the outage? This will let >>>> us know if the flows Neutron setup are getting wiped out. >>>> >>>> On Tue, May 2, 2017 at 12:26 PM, Gustavo Randich < >>>> gustavo.rand...@gmail.com> wrote: >>>> >>>>> Hi Kevin, here is some information aout this issue: >>>>> >>>>> - if the network outage lasts less than ~1 minute, then connectivity >>>>> to host and instances is automatically restored without problem >>>>> >>>>> - otherwise: >>>>> >>>>> - upon outage, "ovs-vsctl show" reports "is_connected: true" in all >>>>> bridges (br-ex / br-int / br-tun) >>>>> >>>>> - after about ~1 minute, "ovs-vsctl show" ceases to show >>>>> "is_connected: true" on every bridge >>>>> >>>>> - upon restoring physical interface (fix outage) >>>>> >>>>> - "ovs-vsctl show" now reports "is_connected: true" in all >>>>> bridges (br-ex / br-int / br-tun) >>>>> >>>>> - access to host and VMs is NOT restored, although some pings >>>>> are sporadically answered by host (~1 out of 20) >>>>> >>>>> >>>>> - to restore connectivity, we: >>>>> >>>>> >>>>> - execute "ifdown br-ex; ifup br-ex" -> access to host is >>>>> restored, but not to VMs >>>>> >>>>> >>>>> - restart neutron-openvswitch-agent -> access to VMs is restored >>>>> >>>>> Thank you! >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Apr 28, 2017 at 5:07 PM, Kevin Benton <ke...@benton.pub> >>>>> wrote: >>>>> >>>>>> With the network down, does ovs-vsctl show that it is connected to >>>>>> the controller? >>>>>> >>>>>> On Fri, Apr 28, 2017 at 2:21 PM, Gustavo Randich < >>>>>> gustavo.rand...@gmail.com> wrote: >>>>>> >>>>>>> Exactly, we access via a tagged interface, which is part of br-ex >>>>>>> >>>>>>> # ip a show vlan171 >>>>>>> 16: vlan171: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc >>>>>>> noqueue state UNKNOWN group default qlen 1 >>>>>>> link/ether 8e:14:8d:c1:1a:5f brd ff:ff:ff:ff:ff:ff >>>>>>> inet 10.171.1.240/20 brd 10.171.15.255 scope global vlan171 >>>>>>> valid_lft forever preferred_lft forever >>>>>>> inet6 fe80::8c14:8dff:fec1:1a5f/64 scope link >>>>>>> valid_lft forever preferred_lft forever >>>>>>> >>>>>>> # ovs-vsctl show >>>>>>> ... >>>>>>> Bridge br-ex >>>>>>> Controller "tcp:127.0.0.1:6633" >>>>>>> is_connected: true >>>>>>> Port "vlan171" >>>>>>> tag: 171 >>>>>>> Interface "vlan171" >>>>>>> type: internal >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 28, 2017 at 3:03 PM, Kevin Benton <ke...@benton.pub> >>>>>>> wrote: >>>>>>> >>>>>>>> Ok, that's likely not the issue then. I assume the way you access >>>>>>>> each host is via an IP assigned to an OVS bridge or an interface that >>>>>>>> somehow depends on OVS? >>>>>>>> >>>>>>>> On Apr 28, 2017 12:04, "Gustavo Randich" <gustavo.rand...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Kevin, we are using the default listen address of loopback >>>>>>>>> interface: >>>>>>>>> >>>>>>>>> # grep -r of_listen_address /etc/neutron >>>>>>>>> /etc/neutron/plugins/ml2/openvswitch_agent.ini:#of_listen_address >>>>>>>>> = 127.0.0.1 >>>>>>>>> >>>>>>>>> >>>>>>>>> tcp/127.0.0.1:6640 -> ovsdb-server >>>>>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info >>>>>>>>> --remote=punix:/var/run/openvswitch/db.sock >>>>>>>>> --private-key=db:Open_vSwitch,SSL,private_key >>>>>>>>> --certificate=db:Open_vSwitch,SSL,certificate >>>>>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir >>>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log >>>>>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Apr 28, 2017 at 5:00 AM, Kevin Benton <ke...@benton.pub> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Are you using an of_listen_address value of an interface being >>>>>>>>>> brought down? >>>>>>>>>> >>>>>>>>>> On Apr 25, 2017 17:34, "Gustavo Randich" < >>>>>>>>>> gustavo.rand...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> (using Mitaka / Ubuntu 16 / Neutron DVR / OVS / VXLAN / >>>>>>>>>>> l2_population) >>>>>>>>>>> >>>>>>>>>>> This sounds very strange (to me): recently, after a switch >>>>>>>>>>> outage, we lost connectivity to all our Mitaka hosts. We had to >>>>>>>>>>> enter via >>>>>>>>>>> iLO host by host and restart networking service to regain access. >>>>>>>>>>> Then >>>>>>>>>>> restart neutron-openvswitch-agent to regain access to VMs. >>>>>>>>>>> >>>>>>>>>>> At first glance we thought it was a problem with the NIC linux >>>>>>>>>>> driver of the hosts not detecting link state correctly. >>>>>>>>>>> >>>>>>>>>>> Then we reproduced the issue simply bringing down physical >>>>>>>>>>> interfaces for around 5 minutes, then up again. Same issue. >>>>>>>>>>> >>>>>>>>>>> And then.... we found that if instead of using native (ryu) >>>>>>>>>>> OpenFlow interface in Neutron Openvswitch we used ovs-ofctl, the >>>>>>>>>>> problem >>>>>>>>>>> disappears. >>>>>>>>>>> >>>>>>>>>>> Any clue? >>>>>>>>>>> >>>>>>>>>>> Thanks in advance. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Mailing list: http://lists.openstack.org/cgi >>>>>>>>>>> -bin/mailman/listinfo/openstack >>>>>>>>>>> Post to : openst...@lists.openstack.org >>>>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi >>>>>>>>>>> -bin/mailman/listinfo/openstack >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
