Thank you Dmitry for very detailed plan and risks assessment. Do we want to run swarm against custom iso with centos7 on Thu evening to measure level of regression? I remember that we were considering this approach.
Regards, Andrey Maximov On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko <dborodae...@mirantis.com > wrote: > With bit more details, I hope this covers all the risks and decision > points now. > > First of all, current list of outstanding commits: > https://etherpad.openstack.org/p/fuel_on_centos7 > > The above list has two sections: backwards compatible changes that can > be merged one at a time even if the rest of CentOS7 support isn't > merged, and backwards incompatible changes that break support for > CentOS6 and must be merged (and, if needed, reverted) all at once. > > Decision point 1: FFE for CentOS7 > > CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it > be allowed a Feature Freeze Exception? So far, the disruption of the > Fuel development process implied by the proposed merge plan is > acceptable, if anything goes wrong and we become unable to have a stable > ISO with merged CentOS7 support on Monday, December 7, the FFE will be > revoked. > > Wed, Dec 2: Merge party > > Merge party before 8.0 FF, we should do our best to merge all remaining > feature commits before end of day (including backwards compatible > CentOS7 support commits), without breaking the build too much. > > At the end of the day we'll start a swarm test over the result of the > merge party, and we expect QA to analyze and summarize the results by > 17:00 MSK (6:00 PST) on Thu Dec 3. > > Risk 1: Merge party breaks the build > > If there is a large regression in swarm pass percentage, we won't be > able to afford a merge freeze which is necessary to merge CentOS7 > support, we'll have to be merging bugfixes until swarm test pass rate is > back around 70%. > > Risk 2: More features get FFE > > If some essential 8.0 features are not completely merged by end of day > Wed Dec 2 and are granted FFE, merging the remaining commits can > interfere with merging CentOS7 support, not just from merge conflicts > perspective, but also invalidating swarm results and making it > practically impossible to bisect and attribute potential regressions. > > Thu, Dec 3: Start merge freeze for CentOS7 > > Decision point 2: Other FFEs > > In the morning MSK time, we will assess Risk 2 and decide what to do > with the other FFEs. The options are: integrate remaining commits into > CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7 > FFE. > > If the decision is to go ahead with CentOS7 merge, we announce merge > freeze for all git repositories that go into Fuel ISO, and spend the > rest of the day rebasing and cleaning up the rest of the CentOS7 commits > to make sure they're all in mergeable state by the end of the day. The > outcome of this work must be a custom ISO image with all remaining > commits, with additional requirement that it must not use Jenkins job > parameters (only patches to fuel-main that change default repository > paths) to specify all required package repositories. This will validate > the proposed fuel-main patches and ensure that no unmerged package > changes are used to produce the ISO. > > Decision point 3: Swarm pass rate > > After swarm results from Wed are available, we will assess the Risk 1. > If the pass rate regression is significant, CentOS7 FFE is revoked and > merge freeze is lifted. If regression is acceptable, we proceed with > merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4. > > Fri, Dec 4: Merge and test CentOS7 > > The team will have until 17:00 MSK to produce a non-custom ISO that > passes BVT and can be run through swarm. > > Sat, Dec 5: Assess CentOS7 swarm and bugfix > > First of all, someone from CI and QA teams should commit to monitoring > the CentOS7 swarm run and report the results as soon as possible. Based > on the results (which once again must be available by 17:00 MSK), we can > decide on the final step of the plan. > > Decision point 4: Keep or revert > > If CentOS7 based swarm shows significant regression, we have to spend > the rest of the weekend including Sunday reverting all CentOS7 commits > that were merged during merge freeze. Once revert is completed, we will > lift the merge freeze. > > If the regression is acceptable, we lift the merge freeze straight away > and proceed with bugfixing as usual. At this point CI team will need to > update the Fuel ISO used for deployment tests in our CI to this same > ISO. > > One way or the other, we will be able to resume bugfixing on Monday > morning MSK time, and will have lost 2 business days (Thu-Fri) during > which we won't be able to merge bugfixes. In addition to that, someone > from QA and everyone from CentOS7 support team has to work on Saturday, > and someone from CI will have to work a few hours on Sunday. > > -- > Dmitry Borodaenko > > > On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote: > > Hello, > > > > We're almost got green BVT on custom CentOS7 ISO and it seems that it's > > the time to discuss the plan how this feature could be merged. > > > > This is not the only one feature that is in a queue. Unfortunately, > > almost any other feature will be broken if merged after CentOS7, so it > > was decided to merge our changes last. > > > > This is not an official announcement, rather a notification letter to > > start a discussion and find any objections. > > > > So the plan is: > > > > * merge all features that are going to be merged before Thusday, Dec 3 > > * call for merge freeze starting at Dec 3, due Dec 7 > > * rebase all CentOS7-related pathsets and resolve any conflicts with > > merged code (Dec 3) > > * build custom ISO, pass BVT (and other tests) (Dec 3) > > * merge all CentOS7-related patchsets at once (Dec 4) > > * build an ISO and pass BVT again (Dec 4) > > * run additional test during weekend (Dec 5, 6) to be sure that ISO > > good enough > > > > According to this plan on Monday, Dec 7 we should either get CentOS7 > > based ISO, or revert all incompatible changes. > > > > -- > > Thanks, > > Dmitry Teselkin > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
