Mike, 2 jobs for Icehouse and Juno equal 2 different repository with packages for Fuel 6.0. This can be problem for current osci workflow. For example: We need building new packages. Which repository we must put packages? to icehouse or/and Juno ? if new packages will break icehouse repository, but required for Juno ...
On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov <mscherba...@mirantis.com> wrote: > Aleksandra, > you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but "4 > fuel-library tests" should happen right after we create stable/5.1. Also, > for Fuel CI for OSTF - I don't think it's actually necessary to support > <5.0 envs. > > Your questions: > > 1. Create jobs for both Icehouse and Juno, but it doesn't make sense > to do staging for Juno till it starts to pass deployment in HA mode. Once > it passes deployment in HA, staging should be enabled. Then, once it passes > OSTF - we extend criteria, and pass only those mirrors which also pass OSTF > phase > 2. Once Juno starts to pass BVT with OSTF check enabled, I think we > can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI > with Icehouse - we might want to continue using them. > > Thanks, > > On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova < > afedor...@mirantis.com> wrote: > >> > Our Fuel CI can do 4 builds against puppet modules: 2 voting, with >> Icehouse packages; 2 non-voting, with Juno packages. >> > Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) >> actually before Juno becomes stable. We will be able to run 2 sets of BVTs >> (against Icehouse and Juno), and it means that we will be able to see >> almost immediately if something in nailgun/astute/puppet integration broke. >> For Juno builds it's going to be all red initially. >> >> Let me rephrase: >> >> We keep one Fuel master branch for two OpenStack releases. And we make >> sure that Fuel master code is compatible with both of them. And we use >> current release (Icehouse) as a reference for test results of upcoming >> release, till we obtain stable enough reference point in Juno itself. >> Moreover we'd like to have OSTF code running on all previous Fuel releases. >> >> Changes to CI workflow look as follows: >> >> Nightly builds: >> 1) We build two mirrors: one for Icehouse and one for Juno. >> 2) From each mirror we build Fuel ISO using exactly the same fuel >> master branch code. >> 3) Then we run BVT tests on both (using the same fuel-main code for >> system tests). >> 4) If Icehouse BVT tests pass, we deploy both ISO images (even with >> failed Juno tests) onto Fuel CI. >> >> On Fuel CI we should run: >> - 4 fuel-library tests (revert master node, inject fuel-library code in >> master node and run deployment): >> 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting >> Juno tests >> - 5 OSTF tests (revert deployed environment, inject OSTF code into >> master node, run OSTF): >> voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on >> master/Juno >> - other tests, which don't use prebuilt environment, work as before >> >> The major action point here would be OSTF tests, as we don't have yet >> working implementation of injecting OSTF code into deployed environment. >> And we don't run any tests on old environments. >> >> >> Questions: >> >> 1) How should we test mirrors? >> >> Current master mirrors go through the 4 hours test cycle involving Fuel >> ISO build: >> 1. we build temporary mirror >> 2. build custom iso from it >> 3. run two custom bvt jobs >> 4. if they pass we move mirror to stable and sitch to it for our >> "primary" fuel_master_iso >> >> Should we test only Icehouse mirrors, or both, but ignoring again failed >> BVT for Juno? Maybe we should enable these tests only later in release >> cycle, say, after SCF? >> >> 2) It is not clear for me when and how we will switch from supporting two >> releases back to one. >> Should we add one more milestone to our release process? The "Switching >> point", when we disable and remove Icehouse tasks and move to Juno >> completely? I guess it should happen before next SCF? >> >> >> >> On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov <mscherba...@mirantis.com >> > wrote: >> >>> > What we need to achieve that is have 2 build series based on Fuel >>> master: one with Icehouse packages, and one with Juno, and, as Mike >>> proposed, keep our manifests backwards compatible with Icehouse. >>> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, >>> with Icehouse packages; 2 non-voting, with Juno packages. >>> >>> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) >>> actually before Juno becomes stable. We will be able to run 2 sets of BVTs >>> (against Icehouse and Juno), and it means that we will be able to see >>> almost immediately if something in nailgun/astute/puppet integration broke. >>> For Juno builds it's going to be all red initially. >>> >>> Another suggestion would be to lower green switch in BVTs for Juno: >>> first, when it passes deployment; and then, if it finally passes OSTF. >>> >>> I'd like to hear QA & DevOps opinion on all the above. Immediately we >>> would need just standard stuff which is in checklists for OSCI & DevOps >>> teams, and ideally soon after that - ability to have Fuel CI running 4 >>> builds, not 2, against our master, as mentioned above. >>> >>> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvya...@mirantis.com> >>> wrote: >>> >>>> All OSCI action items for prepare HCF check list has been done >>>> >>>> >>>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov < >>>> mscherba...@mirantis.com> wrote: >>>> >>>>> Thanks Alexandra. >>>>> >>>>> We land a few patches a day currently, so I think we can open stable >>>>> branch. If we see no serious objections in next 12 hours, let's do it. We >>>>> would need to immediately notify everyone in mailing list - that for every >>>>> patch for 5.1, it should go first to master, and then to stable/5.1. >>>>> >>>>> Is everything ready from DevOps, OSCI (packaging) side to do this? >>>>> Fuel CI, OBS, etc.? >>>>> >>>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova < >>>>> afedor...@mirantis.com> wrote: >>>>> >>>>>> As I understand your proposal, we need to split our HCF milestone >>>>>> into two check points: Branching Point and HCF itself. >>>>>> >>>>>> Branching point should happen somewhere in between SCF and HCF. And >>>>>> though It may coincide with HCF, it needs its own list of requirements. >>>>>> This will give us the possibility to untie two events and make a separate >>>>>> decision on branching without enforcing all HCF criteria. >>>>>> >>>>>> From the DevOps point of view it changes almost nothing, it just adds >>>>>> a bit more discussion items on the management side and slight >>>>>> modifications >>>>>> to our checklists. >>>>>> >>>>>> >>>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko < >>>>>> dborodae...@mirantis.com> wrote: >>>>>> >>>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is >>>>>>> becoming a major problem. No, I don't think we should create >>>>>>> pre-release or feature branches. Instead, we should create stable/5.1 >>>>>>> branches and open master for 6.0 work. >>>>>>> >>>>>>> We have reached a point in 5.1 release cycle where the scope of >>>>>>> issues >>>>>>> we are willing to address in this release is narrow enough to not >>>>>>> require full attention of the whole team. We have engineers working >>>>>>> on >>>>>>> 6.0 features, and their work is essentially blocked until they have >>>>>>> somewhere to commit their changes. >>>>>>> >>>>>>> Simply creating new branches is not even close to solving this >>>>>>> problem: we have a whole CI infrastructure around every active >>>>>>> release >>>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit >>>>>>> commits, package repository mirrors updates, ISO image builds, smoke, >>>>>>> build verification, and swarm tests for ISO images, documentation >>>>>>> builds, etc. A branch without all that infrastructure isn't any >>>>>>> better >>>>>>> than current status quo: every developer tracking their own 6.0 work >>>>>>> locally. >>>>>>> >>>>>>> Unrelated to all that, we also had a lot of very negative experience >>>>>>> with feature branches in the past [0] [1], which is why we have >>>>>>> decided to follow the OpenStack branching strategy: commit all >>>>>>> feature >>>>>>> changes directly to master and track bugfixes for stable releases in >>>>>>> stable/* branches. >>>>>>> >>>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html >>>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html >>>>>>> >>>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF >>>>>>> should remain tied to our ability to declare a release candidate. If >>>>>>> we can't release with the bugs we already know about, declaring HCF >>>>>>> before fixing these bugs would be an empty gesture. >>>>>>> >>>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for >>>>>>> 5.1 will cost us two things: >>>>>>> >>>>>>> 1) DevOps team will have to update our CI infrastructure for one more >>>>>>> release series. It's something we have to do for 6.0 sooner or later, >>>>>>> so this may be a disruption, but not an additional effort. >>>>>>> >>>>>>> 2) All commits targeted for 5.1 will have to be proposed for two >>>>>>> branches (master and stable/5.1) instead of just one (master). This >>>>>>> will require additional effort, but I think that it is significantly >>>>>>> smaller than the cost of spinning our wheels on 6.0 efforts. >>>>>>> >>>>>>> -DmitryB >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov >>>>>>> <dmescherya...@mirantis.com> wrote: >>>>>>> > Hello Fuelers, >>>>>>> > >>>>>>> > Right now we have the following policy in place: the branches for a >>>>>>> > release are opened only after its 'parent' release have reached >>>>>>> hard >>>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 >>>>>>> and >>>>>>> > 6.0. >>>>>>> > >>>>>>> > And that is the problem: if parent release is delayed, we can't >>>>>>> > properly start development of a child release because we don't have >>>>>>> > branches to commit. That is current issue with 6.0: we already >>>>>>> started >>>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes to >>>>>>> > our deployment code we have nowhere to store them. >>>>>>> > >>>>>>> > IMHO the issue could easily be resolved by creation of pre-release >>>>>>> > branches, which are merged together with parent branches once the >>>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial >>>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into >>>>>>> master >>>>>>> > and continue development here. After that pre-6.0 is abandoned. >>>>>>> > >>>>>>> > What do you think? >>>>>>> > >>>>>>> > Thanks, >>>>>>> > >>>>>>> > Dmitry >>>>>>> > >>>>>>> > _______________________________________________ >>>>>>> > OpenStack-dev mailing list >>>>>>> > OpenStack-dev@lists.openstack.org >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dmitry Borodaenko >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Aleksandra Fedorova >>>>>> bookwar >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Scherbakov >>>>> #mihgen >>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> -- >>> Mike Scherbakov >>> #mihgen >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Aleksandra Fedorova >> bookwar >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > Mike Scherbakov > #mihgen > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
