On 4/16/16, 11:59 AM, "Clark Boylan" <cboy...@sapwetik.org> wrote:
>On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote: >> >> >> On 4/15/16, 6:38 PM, "Clark Boylan" <cboy...@sapwetik.org> wrote: >> >> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote: >> >> Hey fellow OpenStackers, >> >> >> >> Kolla 1.0.0 has a fatal flaw in its design related to data >>containers. >> >> The details are outlined here: >> >> >> >>>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.ht >>>>ml >> >> >> >> Our plan to rectify this involves taking what is in stable/mitaka and >> >> making it stable/liberty and placing 1-3 patches on top of >> >>stable/liberty >> >> to make it deploy the liberty version of openstack. These 1-3 >>patches >> >> include metadata like repo locations, upstream tarball locations, >>etc. >> >> No actual code. This is a one time thing; in the future we will be >> >> following a strict bug-backport only policy. >> >> >> >> Our options include: >> >> 1. make a megapatch diff of liberty vs mitaka and merge that, with a >> >> patch on top to fix the repos >> >> Example here: >> >> https://review.openstack.org/#/c/306625/ >> >> >> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete >>the >> >> liberty branch, then create a new stable/liberty branch from the tip >>of >> >> stable/mitaka >> >> >> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and run >>git >> >> reset -hard origin/stable/mitaka outside of gerrit. Tonyb says our >> >>setup >> >> prevents non-fast-forwarded pushes so this might not be viable. >> >> >> >> This was tonyb's work here: >> >> http://paste.openstack.org/raw/494295/ >> >> >> >> What we want to avoid is jamming 1k+ patches through our CI system >>and >> >> having the core reviewers have to deal with acking all those >>patches, or >> >> overloading gerrit and breaking things there. >> >> >> >> I am far from a git wizard, and don't really know the best way to >> >> proceed, other than we must have a liberty 1.1.0 deliverable with >>our >> >> mitaka bits + liberty repo pointers. I'd really like to preserve >> >> history. What we need is for stable/liberty to be an exact >>duplicate of >> >> stable/mitaka codwise, while also preserving history (which option 1 >> >> doesn't do). >> >> >> >> Can the Kolla community get an ack on the options from the >> >>Infrastructure >> >> team one way or another, so I can get the ball rolling on our end and >> >> sync up with Doug on the plan? >> > >> >Is there some reason that the normal backport process used on all of >>the >> >other projects wouldn't work here? Backport the bug fixes you need from >> >the newer branches then release a 1.1.0. >> > >> >Clark >> >> Clark, >> >> Great question. Thanks for the response :) >> >> We have over 1000 patches delta between stable/mitaka and stable/liberty >> and some of them will conflict. But I guess we could do that and jam up >> the gerrit/jenkins with that and the core reviewers with approving all >> those backports. >> >> Unfortunately because of the critical flaw, we really can't just >> cherrypick ~30% of the commits - we need pretty much the whole repo >> intact. Its like a big onion unpeel. In essence we need a docker >> feature >> called named volumes to correct the critical flaw. Before we were using >> data containers which results in data loss. To gain access to named >> volumes with Ansible 1.9.4, we had to write our own docker module called >> kolla_docker.py. kolla_docker.py is different then the Ansible bulit-in >> module to integrate with Docker. The Ansible built-in module only works >> with docker 1.8.2(hard pin) and named volumes were introduced in Docker >> 1.9.0. Ansible Inc has stopped maintenance on Ansible 1.9 series and is >> focusing all of their efforts on Ansible 2.0. Porting our code base to >> Ansible 2.0 is planned for Newton but realistically its a 2-3 month job. >> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their >> Docker >> module - so we still prefer to use our own module to decouple Ansible's >> release cycle from the fast moving Docker upstream (6 y stream releases >>a >> year). >> >> Therefore, kolla-docker is a requirement to remove the hard pin and gain >> access to named volumes. We have approximately 300-500 (super rough >> guess) patches to port our code base to use this kolla_docker.py module. >> We really don't have any consistent way to identify the relevant >>patches. >> I have concerns with this type of backport approach, as we would >> invalidate all of the testing we have done to date with Mitaka and >> Liberty >> - it would basically be forking our own code base for one version of >> Kolla. >> >> It is actually much more complex then I've outlined above - there are >> other things in play here, involving how our diagnostics system works >> (and >> won't work with docker 1.9 and the current Liberty code) meaning we have >> to backport our diagnostics system intact as well. There are other >>parts >> of Kolla that fit into the same category - lots of interdependencies >> mostly centered around kolla_docker.py. >> >> Both Doug and Tony mentioned a backport of this type is not ideal, and a >> better solution should be found. I agree completely - at this point the >> viable options are listed above. >> >> I recognize the above approaches aren't how repos should be managed in >> OpenStack (or anywhere else for that matter). We have corrected our >> deficiencies with backport management this cycle. We are ready to go >> with >> bug-fix only backports from now into the far future because we fixed the >> pain with how Ansible handles Docker integration via kolla-docker.py. >> >> The mission of Kolla is to provide Operators a smooth experience >> deploying >> and operating OpenStack. We would be in violation of our mission in my >> opinion if we did only a partial backport of our repo. >> >> The only other alternative is to abandon the Liberty repository >>entirely, >> which has been discussed. We believe this would prematurely and >> permanently damage the Kolla project's viability in the eyes of >> Operators, >> and result in overall damage to OpenStack in general. >> >> We do plan to tag a 1.1.0 release as soon as one of the above approaches >> is executed - which we think is about 1-2 days of dev work if the repos >> were equivalent. I could see backporting individually using the >>standard >> process taking months and resulting in completely broken software. Our >> best option at this point is to take mitaka as is and point the tarball >> and repo files we depend on at the proper liberty locations. >> >> FWIW the liberty patch has already been produced (against master) - and >> while not complete, results in a working from-source deploy. A second >> patch needs to be developed to make from-binary deploys work. >>Reference: >> >> https://review.openstack.org/#/c/306630/ > >In this case option 2 is probably your best bet. You should be able to >coordinate that with the release team as they have permissions to create >and delete branches on the Kolla project. The basic process would be tag >old branch, abandon all existing changes on stable/liberty, delete the >branch, create the branch with a new head, push and merge the fixes to >switch to liberty things. > >The thing to keep in mind is all of your devs and users will need to >forcefully update their local stable/liberty branches as this will be a >non fastforward change if liberty and mitaka/master have diverged. If >this is going to be a big problem then you should consider the backport >process (either commit by commit or squashed). > >Clark Clark, I don't think this git issue (checking out a new git repo) will be a big problem for our user base but I'll point them at this thread just to make certain. Regards -steve > _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
