On Sat, Apr 16, 2016 at 07:16:27PM +0000, Steven Dake (stdake) wrote: > > > On 4/16/16, 11:59 AM, "Clark Boylan" <cboy...@sapwetik.org> wrote: > > >On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote: > >> > >> > >> On 4/15/16, 6:38 PM, "Clark Boylan" <cboy...@sapwetik.org> wrote: > >> > >> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote: > >> >> Hey fellow OpenStackers, > >> >> > >> >> Kolla 1.0.0 has a fatal flaw in its design related to data > >>containers. > >> >> The details are outlined here: > >> >> > >> > >>>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.ht > >>>>ml > >> >> > >> >> Our plan to rectify this involves taking what is in stable/mitaka and > >> >> making it stable/liberty and placing 1-3 patches on top of > >> >>stable/liberty > >> >> to make it deploy the liberty version of openstack. These 1-3 > >>patches > >> >> include metadata like repo locations, upstream tarball locations, > >>etc. > >> >> No actual code. This is a one time thing; in the future we will be > >> >> following a strict bug-backport only policy. > >> >> > >> >> Our options include: > >> >> 1. make a megapatch diff of liberty vs mitaka and merge that, with a > >> >> patch on top to fix the repos > >> >> Example here: > >> >> https://review.openstack.org/#/c/306625/ > >> >> > >> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete > >>the > >> >> liberty branch, then create a new stable/liberty branch from the tip > >>of > >> >> stable/mitaka > >> >> > >> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and run > >>git > >> >> reset -hard origin/stable/mitaka outside of gerrit. Tonyb says our > >> >>setup > >> >> prevents non-fast-forwarded pushes so this might not be viable. > >> >> > >> >> This was tonyb's work here: > >> >> http://paste.openstack.org/raw/494295/ > >> >> > >> >> What we want to avoid is jamming 1k+ patches through our CI system > >>and > >> >> having the core reviewers have to deal with acking all those > >>patches, or > >> >> overloading gerrit and breaking things there. > >> >> > >> >> I am far from a git wizard, and don't really know the best way to > >> >> proceed, other than we must have a liberty 1.1.0 deliverable with > >>our > >> >> mitaka bits + liberty repo pointers. I'd really like to preserve > >> >> history. What we need is for stable/liberty to be an exact > >>duplicate of > >> >> stable/mitaka codwise, while also preserving history (which option 1 > >> >> doesn't do). > >> >> > >> >> Can the Kolla community get an ack on the options from the > >> >>Infrastructure > >> >> team one way or another, so I can get the ball rolling on our end and > >> >> sync up with Doug on the plan? > >> > > >> >Is there some reason that the normal backport process used on all of > >>the > >> >other projects wouldn't work here? Backport the bug fixes you need from > >> >the newer branches then release a 1.1.0. > >> > > >> >Clark > >> > >> Clark, > >> > >> Great question. Thanks for the response :) > >> > >> We have over 1000 patches delta between stable/mitaka and stable/liberty > >> and some of them will conflict. But I guess we could do that and jam up > >> the gerrit/jenkins with that and the core reviewers with approving all > >> those backports. > >> > >> Unfortunately because of the critical flaw, we really can't just > >> cherrypick ~30% of the commits - we need pretty much the whole repo > >> intact. Its like a big onion unpeel. In essence we need a docker > >> feature > >> called named volumes to correct the critical flaw. Before we were using > >> data containers which results in data loss. To gain access to named > >> volumes with Ansible 1.9.4, we had to write our own docker module called > >> kolla_docker.py. kolla_docker.py is different then the Ansible bulit-in > >> module to integrate with Docker. The Ansible built-in module only works > >> with docker 1.8.2(hard pin) and named volumes were introduced in Docker > >> 1.9.0. Ansible Inc has stopped maintenance on Ansible 1.9 series and is > >> focusing all of their efforts on Ansible 2.0. Porting our code base to > >> Ansible 2.0 is planned for Newton but realistically its a 2-3 month job. > >> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their > >> Docker > >> module - so we still prefer to use our own module to decouple Ansible's > >> release cycle from the fast moving Docker upstream (6 y stream releases > >>a > >> year). > >> > >> Therefore, kolla-docker is a requirement to remove the hard pin and gain > >> access to named volumes. We have approximately 300-500 (super rough > >> guess) patches to port our code base to use this kolla_docker.py module. > >> We really don't have any consistent way to identify the relevant > >>patches. > >> I have concerns with this type of backport approach, as we would > >> invalidate all of the testing we have done to date with Mitaka and > >> Liberty > >> - it would basically be forking our own code base for one version of > >> Kolla. > >> > >> It is actually much more complex then I've outlined above - there are > >> other things in play here, involving how our diagnostics system works > >> (and > >> won't work with docker 1.9 and the current Liberty code) meaning we have > >> to backport our diagnostics system intact as well. There are other > >>parts > >> of Kolla that fit into the same category - lots of interdependencies > >> mostly centered around kolla_docker.py. > >> > >> Both Doug and Tony mentioned a backport of this type is not ideal, and a > >> better solution should be found. I agree completely - at this point the > >> viable options are listed above. > >> > >> I recognize the above approaches aren't how repos should be managed in > >> OpenStack (or anywhere else for that matter). We have corrected our > >> deficiencies with backport management this cycle. We are ready to go > >> with > >> bug-fix only backports from now into the far future because we fixed the > >> pain with how Ansible handles Docker integration via kolla-docker.py. > >> > >> The mission of Kolla is to provide Operators a smooth experience > >> deploying > >> and operating OpenStack. We would be in violation of our mission in my > >> opinion if we did only a partial backport of our repo. > >> > >> The only other alternative is to abandon the Liberty repository > >>entirely, > >> which has been discussed. We believe this would prematurely and > >> permanently damage the Kolla project's viability in the eyes of > >> Operators, > >> and result in overall damage to OpenStack in general. > >> > >> We do plan to tag a 1.1.0 release as soon as one of the above approaches > >> is executed - which we think is about 1-2 days of dev work if the repos > >> were equivalent. I could see backporting individually using the > >>standard > >> process taking months and resulting in completely broken software. Our > >> best option at this point is to take mitaka as is and point the tarball > >> and repo files we depend on at the proper liberty locations. > >> > >> FWIW the liberty patch has already been produced (against master) - and > >> while not complete, results in a working from-source deploy. A second > >> patch needs to be developed to make from-binary deploys work. > >>Reference: > >> > >> https://review.openstack.org/#/c/306630/ > > > >In this case option 2 is probably your best bet. You should be able to > >coordinate that with the release team as they have permissions to create > >and delete branches on the Kolla project. The basic process would be tag > >old branch, abandon all existing changes on stable/liberty, delete the > >branch, create the branch with a new head, push and merge the fixes to > >switch to liberty things. > > > >The thing to keep in mind is all of your devs and users will need to > >forcefully update their local stable/liberty branches as this will be a > >non fastforward change if liberty and mitaka/master have diverged. If > >this is going to be a big problem then you should consider the backport > >process (either commit by commit or squashed). > > > >Clark > > Clark, > > I don't think this git issue (checking out a new git repo) will be a big > problem for our user base but I'll point them at this thread just to make > certain. > Why not highly recommend to your user base to move to mitaka ASAP? Since this will be your latest stable branch.
Seems like a lot of work, and disruption, to liberty for a mitaka release plus 3 patches. > Regards > -steve > > > > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
