Great progress, thank you for keeping everyone informed. On Fri, Dec 4, 2015 at 4:43 PM Dmitry Teselkin <dtesel...@mirantis.com> wrote:
> Hello, > > Status update for Dec, 4 - we're on track. > > As there were no significant issues found during BVT / swarm test runs > on custom CentOS7 ISO it was decided to go with merge party and merge > all CentOS7 related CRs. Our CI was partially updated then and we > started building ISO using production jobs. We've got several green > ISOs which passed BVT. At the moment full set of swarm tests is being > run, we will analyze them tomorrow. > > There are sporadic failures of OSTF tests tied with cinder, we're going > to analyze them tomorrow too. > > On Tue, 1 Dec 2015 13:48:00 -0800 > Dmitry Borodaenko <dborodae...@mirantis.com> wrote: > > > With bit more details, I hope this covers all the risks and decision > > points now. > > > > First of all, current list of outstanding commits: > > https://etherpad.openstack.org/p/fuel_on_centos7 > > > > The above list has two sections: backwards compatible changes that can > > be merged one at a time even if the rest of CentOS7 support isn't > > merged, and backwards incompatible changes that break support for > > CentOS6 and must be merged (and, if needed, reverted) all at once. > > > > Decision point 1: FFE for CentOS7 > > > > CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can > > it be allowed a Feature Freeze Exception? So far, the disruption of > > the Fuel development process implied by the proposed merge plan is > > acceptable, if anything goes wrong and we become unable to have a > > stable ISO with merged CentOS7 support on Monday, December 7, the FFE > > will be revoked. > > > > Wed, Dec 2: Merge party > > > > Merge party before 8.0 FF, we should do our best to merge all > > remaining feature commits before end of day (including backwards > > compatible CentOS7 support commits), without breaking the build too > > much. > > > > At the end of the day we'll start a swarm test over the result of the > > merge party, and we expect QA to analyze and summarize the results by > > 17:00 MSK (6:00 PST) on Thu Dec 3. > > > > Risk 1: Merge party breaks the build > > > > If there is a large regression in swarm pass percentage, we won't be > > able to afford a merge freeze which is necessary to merge CentOS7 > > support, we'll have to be merging bugfixes until swarm test pass rate > > is back around 70%. > > > > Risk 2: More features get FFE > > > > If some essential 8.0 features are not completely merged by end of day > > Wed Dec 2 and are granted FFE, merging the remaining commits can > > interfere with merging CentOS7 support, not just from merge conflicts > > perspective, but also invalidating swarm results and making it > > practically impossible to bisect and attribute potential regressions. > > > > Thu, Dec 3: Start merge freeze for CentOS7 > > > > Decision point 2: Other FFEs > > > > In the morning MSK time, we will assess Risk 2 and decide what to do > > with the other FFEs. The options are: integrate remaining commits into > > CentOS7 merge plan, block remaining commits until Monday, revoke > > CentOS7 FFE. > > > > If the decision is to go ahead with CentOS7 merge, we announce merge > > freeze for all git repositories that go into Fuel ISO, and spend the > > rest of the day rebasing and cleaning up the rest of the CentOS7 > > commits to make sure they're all in mergeable state by the end of the > > day. The outcome of this work must be a custom ISO image with all > > remaining commits, with additional requirement that it must not use > > Jenkins job parameters (only patches to fuel-main that change default > > repository paths) to specify all required package repositories. This > > will validate the proposed fuel-main patches and ensure that no > > unmerged package changes are used to produce the ISO. > > > > Decision point 3: Swarm pass rate > > > > After swarm results from Wed are available, we will assess the Risk 1. > > If the pass rate regression is significant, CentOS7 FFE is revoked and > > merge freeze is lifted. If regression is acceptable, we proceed with > > merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4. > > > > Fri, Dec 4: Merge and test CentOS7 > > > > The team will have until 17:00 MSK to produce a non-custom ISO that > > passes BVT and can be run through swarm. > > > > Sat, Dec 5: Assess CentOS7 swarm and bugfix > > > > First of all, someone from CI and QA teams should commit to monitoring > > the CentOS7 swarm run and report the results as soon as possible. > > Based on the results (which once again must be available by 17:00 > > MSK), we can decide on the final step of the plan. > > > > Decision point 4: Keep or revert > > > > If CentOS7 based swarm shows significant regression, we have to spend > > the rest of the weekend including Sunday reverting all CentOS7 commits > > that were merged during merge freeze. Once revert is completed, we > > will lift the merge freeze. > > > > If the regression is acceptable, we lift the merge freeze straight > > away and proceed with bugfixing as usual. At this point CI team will > > need to update the Fuel ISO used for deployment tests in our CI to > > this same ISO. > > > > One way or the other, we will be able to resume bugfixing on Monday > > morning MSK time, and will have lost 2 business days (Thu-Fri) during > > which we won't be able to merge bugfixes. In addition to that, someone > > from QA and everyone from CentOS7 support team has to work on > > Saturday, and someone from CI will have to work a few hours on Sunday. > > > > > > -- > Thanks, > Dmitry Teselkin > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Mike Scherbakov #mihgen
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
