Hi Paul, I just hope you won’t reinvent the wheel ;-) Feel free to use what was build to test the 500+ PRs that got merged over the last couple of months to build 4.6, 4.7 and 4.8. For results see the comments in all these PRs.
More info: https://github.com/schubergphilis/MCT-shared This is already being used by PCExtreme, Leaseweb and Schuberg Philis. Regards, Remi On 19/01/16 22:01, "Paul Angus" <paul.an...@shapeblue.com> wrote: >Hi Ilya (and all others), > >We (ShapeBlue) agree with you regarding the importance of automated >integration testing. The consulting team are currently working to understand >Marvin fully and what pieces we need to be able deploy properly representative >(virtualised) infrastructures to test against. We intend to open source our >resultant framework, with a view to community members being able to use it >themselves in their own labs but also so that as a community we can build it >out somewhere to be used as part of the projects' testing regime. > >We'll be reaching out to the community soon to work on the failures we're >seeing currently and understand where they come from - ie Marvin build, the >environment, the tests or CloudStack (watch out Mike T - I'm about to watch >your presentation video). > >We see this an important initiative for CloudStack as a whole, agile or >otherwise. But ultimately this is 'only' regression testing and the >organisations which need/want LTS, require it for a number of reasons, >including the fact new features bring new bugs which, if we knew we had to >test for, probably wouldn't have been there in the first place. > > > >Paul Angus >VP Technology , ShapeBlue > > >t: @cloudyangus<tel:@cloudyangus> > >e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> | > w: www.shapeblue.com<http://www.shapeblue.com> > > > > > >-----Original Message----- >From: ilya [mailto:ilya.mailing.li...@gmail.com] >Sent: Tuesday, January 19, 2016 6:39 PM >To: dev@cloudstack.apache.org >Subject: Re: [PROPOSAL] LTS Release Cycle > >> Therefore, the process should strive to make as a few releases as >necessary to achieve this goal. > >I guess part two to this question would be - we need the automated testing >environments. This can ensure rapid release testing and acutal release, and we >dont have to restrains ourselves to limited number of releases LTS release due >to QA pains. > >We need a separate initiative with CloudStack testing framework. > >On 1/18/16 6:54 PM, John Burwell wrote: >> Ilya, >> >> Unless we have a bug fix that addresses a significant, widespread system >> stability problem or a high priority/impact security issue, an LTS will roll >> up a number of fixes. Each release would receive the full system test to >> verify that the patch set does not introduce regression defects. I believe >> that most LTS users want a few releases as necessary to keep their systems >> up-to-date and stable because each upgrade carries operational risk and >> downtime. Therefore, the process should strive to make as a few releases as >> necessary to achieve this goal. >> >> Thanks, >> -John >> >>> On Jan 15, 2016, at 3:22 PM, ilya <ilya.mailing.li...@gmail.com> wrote: >>> >>> John >>> >>> Thank you for taking time writing out the LTS proposal. >>> >>>> Broad community support is vital to guarantee the twenty (20) month >>>> support period for each LTS branch. Given the ebbs and flows of >>>> contribution and committer priorities, ShapeBlue will provide a >>>> release manager, as well as, engineering support to fill any >>>> contribution gaps to ensure that the community fulfills LTS commitments. >>> >>> You guys rock!! >>> >>> I'm +1 on this, >>> >>> Can you please expand on the QA side of LTS. Since this is more >>> around long term bug/security fix - i'd think - the testing will be >>> minimal, to the scope that fix applies - which will speed up the >>> release process in general. What are your thoughts on this? >>> >>> >>> Thanks >>> ilya >>> >>> >>> >>> >>> >>> On 1/15/16 10:48 AM, John Burwell wrote: >>>> Motivation >>>> ======== >>>> >>>> The current monthly release cycle addresses the needs of users >>>> focused on deploying new functionality as quickly as possible. It >>>> does not address the needs of users oriented towards stability >>>> rather than new functionality. These users typically employ QA >>>> processes to comply with corporate policy and/or regulatory >>>> requirements. To maintain a growing, thriving community, we must address >>>> the needs of both user types. >>>> Therefore, I propose that we overlay a LTS release cycle onto the >>>> monthly release cycle to address the needs of stability-oriented >>>> users with minimal to no impact on the monthly release cycle. This >>>> proposed LTS release cycle has the following goals: >>>> >>>> * Prefer Stability to New Functionality: Deliver releases that only >>>> address defects and CVEs. This narrowly focused change scope greatly >>>> reduces the upgrade risk/operational impact and shorter internal QA cycles. >>>> * Reliable Release Lifetimes: Embracing a time-based release >>>> strategy, the LTS release cycle will provide users with a reliable >>>> support time frames. Users can use these time frames provide users >>>> with an 20 month window in which to plan upgrades. >>>> * Support Sustainability: With a defined end of support for LTS >>>> releases and a maximum of two (2) LTS releases under active >>>> maintenance at any given time, community members can better plan >>>> their commitments to release support activities. We also have a >>>> agreed upon policy for release end-of-life (EOL) to debate about >>>> continuing work on old releases. >>>> >>>> Proposed Process >>>> ============== >>>> >>>> LTS release branches will be cut twice year on 1 Jan and 1 July from >>>> the tag of the most recent monthly release. The branch will be named >>>> <base >>>> version>-LTS and each LTS release will be versioned in the form of >>>> version><base -<LTS revision number>. For example, if we cut an LTS >>>> version>branch >>>> based on 4.7.0, the branch would be named 4.7.0-LTS and the version >>>> of the first LTS release would be 4.7.0-0, the second would be >>>> 4.7.0-1, etc. This release naming convention differentiates LTS and >>>> monthly releases, communicates the version on which the LTS release >>>> is based, and allows the maintenance releases for monthly releases >>>> without version number contention/conflict. Finally, like master, an >>>> LTS branch would be always deployable following its initial release. >>>> While it is unlikely that LTS users would deploy from the branch, >>>> the quality discipline of this requirement will benefit the long term >>>> stability of LTS releases. >>>> All PRs targeting an LTS would require two LGTMs in order to be merged. >>>> >>>> The following are the types of changes that would permitted and >>>> guarantees provided to users: >>>> >>>> * No features or enhancements would be backported to LTS release branches. >>>> * Database changes would be limited to those required to address the >>>> backported defect fixes. >>>> * Support for the release/version of the following components from >>>> the release on which the LTS is based throughout the entire release cycle: >>>> * MySQL/MariaDB >>>> * JDK/JRE >>>> * Linux distributions >>>> * API compatibility for between all LTS revisions. API changes would >>>> be limited to those required to fix defects or address security issues. >>>> >>>> An LTS release would have a twenty (20) month lifetime from the date >>>> the release branch is cut. This support period allows up to two (2) >>>> months of branch stabilization before initial release with a minimum >>>> of eighteen (18) months of availability for deployment. LTS releases >>>> would have the following support periods: >>>> >>>> * 0-14 months: backport all defect fixes in the scope of the LTS >>>> release functionality; fix all blocker and critical priority defects >>>> identified in the branch >>>> * 14-20 months: backport only CVE fixes in the scope of the LTS >>>> release functionality; fix all blocker priority defects in the >>>> branch >>>> >>>> Defect fixes that originate in an LTS branch will be pulled forward >>>> to master only. Finally, an LTS branch should be release the fewest >>>> times necessary to deliver fixes in a relatively timely manner. >>>> Therefore, LTS releases will be triggered based on the number of >>>> pending of fixes and their severity with no defect fix awaiting >>>> release more than forty-five >>>> (45) days. CVE fixes would trigger an immediate release of an LTS branch. >>>> >>>> Resourcing and Proposed Timeline >>>> =========================== >>>> >>>> Broad community support is vital to guarantee the twenty (20) month >>>> support period for each LTS branch. Given the ebbs and flows of >>>> contribution and committer priorities, ShapeBlue will provide a >>>> release manager, as well as, engineering support to fill any >>>> contribution gaps to ensure that the community fulfills LTS commitments. >>>> >>>> In order to prepare for supporting LTS releases, we would need to >>>> complete the following items: >>>> >>>> 1. All tools that do version number comparisons would be made aware >>>> of the LTS versioning scheme 2. Officially support running the >>>> management server on Java 8 since Java >>>> 7 has been EOL since last April [1] (i.e. compile to 1.7 with the >>>> Java >>>> 1.8 compiler and run on Java 1.8). Providing a 20-month support >>>> window on an EOL JVM is an unacceptable security risk. >>>> 3. Update the wiki and website to explain the new release cycle and >>>> help users decide which release type suits their needs 4. Define an >>>> initial test plan for LTS stabilization 5. Agree on the definitions >>>> of ticket severities >>>> >>>> In order to address these items and start on a regular rhythm, I >>>> propose that first LTS cycle begin on 1 July 2015. In the interim, >>>> we would continue to ship critical bug fixes from the 4.5 release as >>>> this release seems to be the most commonly deployed in the community. >>>> >>>> Together, the monthly and LTS release cycles address the needs of >>>> the entire Apache CloudStack user community. I believe that the >>>> process described in the proposal will yield releases that meet the >>>> needs of users requiring release stability without adversely >>>> affecting the velocity of the monthly release cycle. >>>> >>>> Thanks, >>>> -John >>>> >>>> [1]: http://www.oracle.com/technetwork/java/eol-135779.html >>>> >>>> >>>> ShapeBlue <http://www.shapeblue.com> John Burwell ShapeBlue >>>> >>>> d: *+44 (20) 3603 0542 | s: +1 (571) 403-2411 * >>>> <tel:+44%20(20)%203603%200542%20|%20s:%20+1%20(571)%20403-2411> >>>> >>>> e: *john.burw...@shapeblue.com | t: * >>>> <mailto:john.burw...@shapeblue.com%20|%20t:> | w: >>>> *www.shapeblue.com* <http://www.shapeblue.com> >>>> >>>> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK >>>> >>>> Shape Blue Ltd is a company incorporated in England & Wales. >>>> ShapeBlue Services India LLP is a company incorporated in India and >>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil >>>> Consultoria Ltda is a company incorporated in Brasil and is operated >>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company >>>> registered by The Republic of South Africa and is traded under >>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >>>> This email and any attachments to it may be confidential and are >>>> intended solely for the use of the individual to whom it is addressed. >>>> Any views or opinions expressed are solely those of the author and >>>> do not necessarily represent those of Shape Blue Ltd or related companies. >>>> If you are not the intended recipient of this email, you must >>>> neither take any action based upon its contents, nor copy or show it to >>>> anyone. >>>> Please contact the sender if you believe you have received this >>>> email in error. >>>> >>>> >>>> Find out more about ShapeBlue and our range of CloudStack related services: >>>> IaaS Cloud Design & Build >>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – >>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/> >>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> >>>> | CloudStack Software Engineering >>>> <http://shapeblue.com/cloudstack-software-engineering/> >>>> CloudStack Infrastructure Support >>>> <http://shapeblue.com/cloudstack-infrastructure-support/> | >>>> CloudStack Bootcamp Training Courses >>>> <http://shapeblue.com/cloudstack-training/> >> >> Find out more about ShapeBlue and our range of CloudStack related services: >> IaaS Cloud Design & >> Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – >> rapid IaaS deployment framework<http://shapeblue.com/csforge/> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | >> CloudStack Software >> Engineering<http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure >> Support<http://shapeblue.com/cloudstack-infrastructure-support/> | >> CloudStack Bootcamp Training >> Courses<http://shapeblue.com/cloudstack-training/> >> >Find out more about ShapeBlue and our range of CloudStack related services: >IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> >| CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | >CloudStack Software >Engineering<http://shapeblue.com/cloudstack-software-engineering/> >CloudStack Infrastructure >Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack >Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
