On Mon, Feb 29, 2016 at 01:19:29PM +0300, Vladimir Kuklin wrote: > Thanks for bringing this up. Frankly, I think that we hurried a little bit > by making our integration with upstream puppet manifests too continuous. I > would suppose that we used a little bit different technique.
I don't think "hurried" is applicable here: the only way to become more ready to track upstream than we already are is to *start* tracking upstream. Postponing that leaves us in a Catch-22 situation where we can't stay in sync with upstream because we're not continuously catching up with upstream. > First of all, we need to have a set of stable Fuel commits against which > the changes to upstream manifests should be tested. Could you please > elaborate on whether we are doing this already? We are using the same known-good Fuel ISO for puppet-openstack and fuel-library, so in that sense, yes, we are doing this already. We use the HEAD of master branch of fuel-library in these tests, because using the fuel-library commit from the known-good ISO would mean having to update that ISO every time we merge a commit that brings fuel-library up to date with recent changes in puppet-openstack. > Secondly, we need to have FUEL CI working with a set of stable commits of > puppet openstack manifests which passed those upstream tests as we should > not have too much moving parts or we will get into situation similar to > requirements.txt updates when we have pypi updated with new library, e.g. > oslo-serialization. That would lock us into that Catch-22 situation where we can't allow Fuel CI to vote on puppet-openstack commits because fuel-library is always too far behind puppet-openstack for its votes to mean anything useful. We have to approach this from the opposite direction: make Fuel CI stable and meaningful enough so that, 9 times out of 10, Fuel CI failure indicates a real problem with the code, and the remaining cases can be quickly unblocked by pushing a catch-up commit to fuel-library (ideally with a Depends-On tag). It is a matter of trust between projects: do we trust Puppet OpenStack project to take Fuel's problems seriously and to avoid breaking our CI more often than necessary? Can Puppet OpenStack project trust us with the same? So far, our collaboration track record has been pretty good bordering on examplary, and I think we should build on that and move forward instead of clinging to the old ways. > In this case, we will be able to do proper testing against frozen code-base > for each piece thus avoiding such issues while retaining fair amount of > integration with upstream puppet manifests for OpenStack. The problem with moving only one piece at a time is that you end up so far behind that you're slowing everyone down. BKL and GIL are not the only way to deal with concurrency, we can do better than that. -- Dmitry Borodaenko > On Fri, Feb 26, 2016 at 1:28 PM, Ivan Berezovskiy <iberezovs...@mirantis.com > > wrote: > > > Hello, Fuelers! > > > > Yesterday we've faced an issue which came from puppet-neutron > > module: LP #1549934 <https://bugs.launchpad.net/fuel/+bug/1549934>. Fix > > was prepared very fast: > > https://review.openstack.org/#/c/284882/ (thanks Sergey for this). > > So, If CI is red on your patch please re-base it on top of master. > > > > Anyway, this issue affected a lot of patches and blocked some developers, > > because BVT and neutron_smoke tests was also broken. We need to find > > a way how to minimize risks and affection of such changes on fuel-library. > > We have jobs which monitors upstream patches: > > https://ci.fuel-infra.org/view/puppet-openstack/ > > Let's start to monitor those jobs on daily basis. We should have at least 1 > > (ideally 2 or more) engineers which are responsible for analysis of those > > CI failures. If patch to puppet module is incorrect - we should review it > > with explanation what is actually wrong. If patch is correct, but breaks > > current Fuel CI, it means that problem is in our side and we should prepare > > fuel-library adapt patch to fix the issue. Ideally, we should have an > > ability > > to test this fuel-library patch together with upstream one e.g. using > > 'Depends on' > > in commit message. > > > > Thoughts? > > > > -- > > Thanks, Ivan Berezovskiy > > MOS Puppet Team Lead > > at Mirantis <https://www.mirantis.com/> > > > > slack: iberezovskiy > > skype: bouhforever > > phone: + 7-960-343-42-46 > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > -- > Yours Faithfully, > Vladimir Kuklin, > Fuel Library Tech Lead, > Mirantis, Inc. > +7 (495) 640-49-04 > +7 (926) 702-39-68 > Skype kuklinvv > 35bk3, Vorontsovskaya Str. > Moscow, Russia, > www.mirantis.com <http://www.mirantis.ru/> > www.mirantis.ru > vkuk...@mirantis.com > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
