Andrew, [2] may be due to agents failing to start. Incorrect agents configuration will lead to agents unability to start and pacemaker timeouts. [3] I do not see failures in rabbit service as I see that it succesfully transitioned from stopped to running state. [4] Swift error shows that you have incorrect rings configuration.
On Tue, Jul 15, 2014 at 9:23 AM, Andrew Woodward <xar...@gmail.com> wrote: > Friday, resolved the haproxy / vip issue. It was due to loosing > net.ipv4.ip_forward [1] and then we regressed back to not working at > all in HA or Simple. There was also an issue with the neutron keystone > call being cached oddly causing a trace. Removing the ability to cache > the object appears to have stopped the error, but the changes leading > into learning about this issue may have steered us into this not > working again. > > Today, fought on the regression > > HA deployment testing is completely blocked by [2], [3]. Usually this > occurs after puppet re-runs the controller after an error. This > frequently occurs due to swift [4]. > > Simple deployment testing is plagued by neutron notifiers auth failures > [5]. > > There is some minor annoyance with [6] that doesn't have to be fixed, > but is annoying. There is another variant where other, non-recoverable > (usually malformed-syntax errors) are also caught in the retry when > they should be outright aborted on. This is likely due to the regex in > neutron.rb being overly grabby. > > [1] https://bugs.launchpad.net/fuel/+bug/1340968 > [2] > https://gist.github.com/xarses/219be742ab04faeb7f53#file-pacemaker-corosync-error > [3] > https://gist.github.com/xarses/219be742ab04faeb7f53#file-rabbit-service-failure > [4] https://gist.github.com/xarses/219be742ab04faeb7f53#file-swift-error > [5] > https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron-vif-plug-error > [6] > https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron_network-changes-shared-on-existing-network > > On Fri, Jul 11, 2014 at 12:08 AM, Andrew Woodward <xar...@gmail.com> > wrote: > > Retested today > > ubuntu single nova vlan - works > > centos single nova dhcp - works > > ubuntu single neutron gre - works > > centos single neutron vlan - works > > centos ha(1) neutron vlan - fail haproxy issue > > ubuntu ha(1) neutron gre - fail haproxy issue. > > > > haproxy / vip issue: > > > > due to whatever reason that I haven't been able to track down yet, the ip > > netns namespaces (public and management) ns_IPaddr2 vips can not ping or > > otherwise communicate with nodes remote to who ever owns the respective > vip. > > Once this issue is resolved, I believe that CI should pass given that the > > build appears 100% functional except that computes cant connect to the > vip > > properly. > > > > > > On Thu, Jul 10, 2014 at 1:05 AM, Mike Scherbakov < > mscherba...@mirantis.com> > > wrote: > >> > >> We had a call between Andrew (@xarses), Vladimir Kuklin (@aglarendil) > and > >> myself (@mihgen) today to finally sort out Neutron ML2 integration in > Fuel. > >> We didn't have @xenolog on the call, but hopefully he is more or less > fine > >> with all below, and kindly request him to confirm :) > >> Discussed following topics, with an agreement from all participants: > >> > >> Risks of merging https://review.openstack.org/#/c/103280 (@xarses, > >> upstream puppet module, will further refer as "280") vs > >> https://review.openstack.org/#/c/103947 (@xenolog, extending existing > puppet > >> module with ML2 support, will further refer as "947") > >> > >> We all agree, that 280 is strategically the way to go. It was so by > >> design, and 947 was done only as risk mitigation plan in case if 280 is > not > >> ready in time > >> Both 280 and 947 were manually verified in combinations of > >> ubuntu/centos/vlan/gre/ha, 280 needs to be verified with nova-network > >> 947 was ready a week ago and considered to be more stable solution > >> 280 is has much higher risks to introduce regressions, as it is > basically > >> rearchitecturing of Neutron puppet module in Fuel > >> 947 has backward compatibility support with legacy OVS, while 280 > doesn't > >> have it at the moment > >> > >> Mellanox & VMWare NSX dependency on ML2 implementation rebase time > >> > >> Rebase itself should not be hard > >> It has to be tested and may take up to next WE to do all > >> rebases/testing/fixing > >> As both Mellanox & NSX Neutron pieces are isolated, it can be an > exception > >> for merging by next TH > >> > >> Discussed sanitize_network_config [1] > >> > >> @aglarendil points out that we need to verify input params which puppet > >> receives in advance, before waiting hours of deploy > >> @mihgen's point of view is that we need to consider each Fuel component > as > >> module, and verify output of it with certain input params. So there is > no > >> point to verify input in puppet, if it's being verified in output of > >> Nailgun. > >> @xarses says that we need to verify configuration files created in > system > >> after module execution, to check if it matches with module input params > >> Overall topic has been discussed much more extensively, and it needs > >> further follow up. @aglarendil confirms that it is Ok to remove > sanitize for > >> now, but start writing much more tests to check that 280 deploys > correctly > >> right away > >> > >> Action plan we've come up with: > >> > >> Merge 947, as less risky at the moment, also considering the fact that > 280 > >> doesn't pass CI > >> > >> this will immediately unblock Mellanox & NSX teams so they can rebase on > >> ML2 implementation > >> > >> In parallel, create a test plan for 280 together with QA team > >> @xenolog to join 280's effort, start fixing issues discovered there, > >> including [2] > >> Build an ISO, based on 280, and start testing it according to the plan > >> Merge 280 once test plan is executed and issues fixed. Expected on > Monday. > >> > >> [1] https://review.openstack.org/#/c/99807/1/specs/5.1/ml2-neutron.rst > >> [2] > >> > https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/1608/ > >> > >> Thanks, > >> > >> > >> On Wed, Jul 9, 2014 at 10:31 PM, Andrew Woodward <xar...@gmail.com> > wrote: > >>>> > >>>> Teams from MElanox and VMware based their work on current > implementation > >>>> of Neutron > >>> > >>> Mlnx appears to not set any neutron settings, so wont be enabled > >>> correctly with out some more TLC > >>> ( > >>> > https://wiki.openstack.org/wiki/Mellanox-Neutron-Icehouse-Redhat#Neutron_Server_Node > >>> ) > >>> > >>> NSX appears to be using legacy OVS, but assumes just whatever the > default > >>> core_plugin is so it will need some TLC too. > >>> > >>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1 > >>> > >>> Link is wrong should be https://review.openstack.org/#/c/103947 > >>> > >>>> > >>>> To base Neutron implementation for 6.0 get > >>>> https://review.openstack.org/#/c/103280/ and start working for adapt > this > >>>> for Juno Neutron. > >>> > >>> I still think we should merge it > >>>> > >>>> Also we should discuss how to make HA-wrapper more pluggable. > >>> > >>> https://review.openstack.org/#/c/103279/ makes it very pluggable, and > >>> felt like the best start, what else do we need to add in the near term? > >>> Should we just extend it as we move more components to it? > >>> > >>> > >>> On Wed, Jul 9, 2014 at 9:21 AM, Sergey Vasilenko > >>> <svasile...@mirantis.com> wrote: > >>>> > >>>> After review https://review.openstack.org/#/c/103280/ I have > following > >>>> thoughts. This patch looks fine, but huge and has lot of changes in > >>>> deployment logic. Parallel teams work (vmware, melanox) depends on > current > >>>> implementation of Neutron in FUEL. I have a some dreads, that they > can’t > >>>> refactor his code in the short time remaining. > >>>> > >>>> Implementation https://review.openstack.org/#/c/103280 has > considerably > >>>> less changes in current code of FUEL. Teams from MElanox and VMware > based > >>>> their work on current implementation of Neutron. Also this > implementation > >>>> was already tested in most deployment modes. > >>>> > >>>> On this basis, I suggest: > >>>> > >>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1 > >>>> To base Neutron implementation for 6.0 get > >>>> https://review.openstack.org/#/c/103280/ and start working for adapt > this > >>>> for Juno Neutron. > >>>> Also we should discuss how to make HA-wrapper more pluggable. > >>>> > >>>> > >>>> _______________________________________________ > >>>> OpenStack-dev mailing list > >>>> OpenStack-dev@lists.openstack.org > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>> > >>> > >>> > >>> -- > >>> Andrew > >>> Mirantis > >>> Ceph community > >>> > >>> _______________________________________________ > >>> OpenStack-dev mailing list > >>> OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> > >> > >> > >> > >> -- > >> Mike Scherbakov > >> #mihgen > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > > > -- > > Andrew > > Mirantis > > Ceph community > > > > -- > Andrew > Mirantis > Ceph community > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com <http://www.mirantis.ru/> www.mirantis.ru vkuk...@mirantis.com
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
