Dmitry, thank you! I confirm that the plan looks good for our team, we'll follow it.
Regards, Igor Marnat On Wed, Dec 2, 2015 at 9:39 PM, Andrew Maksimov <amaksi...@mirantis.com> wrote: > Thank you Dmitry for very detailed plan and risks assessment. > Do we want to run swarm against custom iso with centos7 on Thu evening to > measure level of regression? I remember that we were considering this > approach. > > Regards, > Andrey Maximov > > > On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko < > dborodae...@mirantis.com> wrote: > >> With bit more details, I hope this covers all the risks and decision >> points now. >> >> First of all, current list of outstanding commits: >> https://etherpad.openstack.org/p/fuel_on_centos7 >> >> The above list has two sections: backwards compatible changes that can >> be merged one at a time even if the rest of CentOS7 support isn't >> merged, and backwards incompatible changes that break support for >> CentOS6 and must be merged (and, if needed, reverted) all at once. >> >> Decision point 1: FFE for CentOS7 >> >> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it >> be allowed a Feature Freeze Exception? So far, the disruption of the >> Fuel development process implied by the proposed merge plan is >> acceptable, if anything goes wrong and we become unable to have a stable >> ISO with merged CentOS7 support on Monday, December 7, the FFE will be >> revoked. >> >> Wed, Dec 2: Merge party >> >> Merge party before 8.0 FF, we should do our best to merge all remaining >> feature commits before end of day (including backwards compatible >> CentOS7 support commits), without breaking the build too much. >> >> At the end of the day we'll start a swarm test over the result of the >> merge party, and we expect QA to analyze and summarize the results by >> 17:00 MSK (6:00 PST) on Thu Dec 3. >> >> Risk 1: Merge party breaks the build >> >> If there is a large regression in swarm pass percentage, we won't be >> able to afford a merge freeze which is necessary to merge CentOS7 >> support, we'll have to be merging bugfixes until swarm test pass rate is >> back around 70%. >> >> Risk 2: More features get FFE >> >> If some essential 8.0 features are not completely merged by end of day >> Wed Dec 2 and are granted FFE, merging the remaining commits can >> interfere with merging CentOS7 support, not just from merge conflicts >> perspective, but also invalidating swarm results and making it >> practically impossible to bisect and attribute potential regressions. >> >> Thu, Dec 3: Start merge freeze for CentOS7 >> >> Decision point 2: Other FFEs >> >> In the morning MSK time, we will assess Risk 2 and decide what to do >> with the other FFEs. The options are: integrate remaining commits into >> CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7 >> FFE. >> >> If the decision is to go ahead with CentOS7 merge, we announce merge >> freeze for all git repositories that go into Fuel ISO, and spend the >> rest of the day rebasing and cleaning up the rest of the CentOS7 commits >> to make sure they're all in mergeable state by the end of the day. The >> outcome of this work must be a custom ISO image with all remaining >> commits, with additional requirement that it must not use Jenkins job >> parameters (only patches to fuel-main that change default repository >> paths) to specify all required package repositories. This will validate >> the proposed fuel-main patches and ensure that no unmerged package >> changes are used to produce the ISO. >> >> Decision point 3: Swarm pass rate >> >> After swarm results from Wed are available, we will assess the Risk 1. >> If the pass rate regression is significant, CentOS7 FFE is revoked and >> merge freeze is lifted. If regression is acceptable, we proceed with >> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4. >> >> Fri, Dec 4: Merge and test CentOS7 >> >> The team will have until 17:00 MSK to produce a non-custom ISO that >> passes BVT and can be run through swarm. >> >> Sat, Dec 5: Assess CentOS7 swarm and bugfix >> >> First of all, someone from CI and QA teams should commit to monitoring >> the CentOS7 swarm run and report the results as soon as possible. Based >> on the results (which once again must be available by 17:00 MSK), we can >> decide on the final step of the plan. >> >> Decision point 4: Keep or revert >> >> If CentOS7 based swarm shows significant regression, we have to spend >> the rest of the weekend including Sunday reverting all CentOS7 commits >> that were merged during merge freeze. Once revert is completed, we will >> lift the merge freeze. >> >> If the regression is acceptable, we lift the merge freeze straight away >> and proceed with bugfixing as usual. At this point CI team will need to >> update the Fuel ISO used for deployment tests in our CI to this same >> ISO. >> >> One way or the other, we will be able to resume bugfixing on Monday >> morning MSK time, and will have lost 2 business days (Thu-Fri) during >> which we won't be able to merge bugfixes. In addition to that, someone >> from QA and everyone from CentOS7 support team has to work on Saturday, >> and someone from CI will have to work a few hours on Sunday. >> >> -- >> Dmitry Borodaenko >> >> >> On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote: >> > Hello, >> > >> > We're almost got green BVT on custom CentOS7 ISO and it seems that it's >> > the time to discuss the plan how this feature could be merged. >> > >> > This is not the only one feature that is in a queue. Unfortunately, >> > almost any other feature will be broken if merged after CentOS7, so it >> > was decided to merge our changes last. >> > >> > This is not an official announcement, rather a notification letter to >> > start a discussion and find any objections. >> > >> > So the plan is: >> > >> > * merge all features that are going to be merged before Thusday, Dec 3 >> > * call for merge freeze starting at Dec 3, due Dec 7 >> > * rebase all CentOS7-related pathsets and resolve any conflicts with >> > merged code (Dec 3) >> > * build custom ISO, pass BVT (and other tests) (Dec 3) >> > * merge all CentOS7-related patchsets at once (Dec 4) >> > * build an ISO and pass BVT again (Dec 4) >> > * run additional test during weekend (Dec 5, 6) to be sure that ISO >> > good enough >> > >> > According to this plan on Monday, Dec 7 we should either get CentOS7 >> > based ISO, or revert all incompatible changes. >> > >> > -- >> > Thanks, >> > Dmitry Teselkin >> > >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
