Hi Will, We used this to test hundreds of PRs: https://github.com/schubergphilis/MCT-shared/. I can talk you through it sometime next week or so if you want.
Regards, Remi On 01/03/16 21:50, "williamstev...@gmail.com on behalf of Will Stevens" <williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote: >Thanks Paul. Great to hear you are planning to make everything available >and you are willing to take contributions from the community. > >I am currently trying to get up to speed on everything that has been done >and all the different pieces in play. I would like to have a CI in place >(even if it is only locally on my hardware) for testing PRs for 4.9. My >focus right now is solving the CI problem, so I am interested in any >developments on this front. > >We have Travis doing smoke tests on every PR right now. Travis is free for >open source projects. Is there a reason we are not doing more extensive >tests with Travis since it is already integrated into github? Like maybe >running the simulator on every PR? I think this could be a good first step >to weed out some of the PRs that are likely to fail a full CI run. > >Cheers, > >*Will STEVENS* >Lead Developer > >*CloudOps* *| *Cloud Solutions Experts >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >w cloudops.com *|* tw @CloudOps_ > >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <paul.an...@shapeblue.com> wrote: > >> Hi Will, >> >> It will be 100% open source. The $dayjob has blocked the team from working >> on it recently. I'd like to get some of our ideas into the CloudStack wiki >> this week before I go on leave for a week. >> Rohit has created a repo for us: https://github.com/shapeblue/Trillian >> (which looks a bit sad at the moment - I hope to push our Ansible roles >> for building mgmt. servers and Marvin boxes into it this week also) >> >> Internally we're planning a two week sprint on it starting 14th March. >> Having to dip in and out of this effort is making it very difficult to get >> anywhere >> >> I think the biggest part of the work will be to get Marvin up to scratch, >> the log output from the tests is pretty bad. >> >> We very much want this to be a community 'thing', and we're happy to work >> with everyone/anyone to get there. >> >> Any ideas on the most effective way to have a number of people on this - >> I'm all ears. >> In the meantime I'll try to make good on my promises to get our ideas >> visible to everyone... >> >> >> >> >> Paul Angus >> VP Technology , ShapeBlue >> >> >> t: @cloudyangus<tel:@cloudyangus> >> >> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> >> | w: www.shapeblue.com<http://www.shapeblue.com> >> >> >> >> >> >> -----Original Message----- >> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On >> Behalf Of Will Stevens >> Sent: Monday, February 29, 2016 9:52 PM >> To: dev@cloudstack.apache.org >> Subject: Re: Important Pending Items >> >> @Paul and @Bharat, thank you for the quality write-ups on this topic. >> >> re:Paul's Details: >> I share a lot of your thoughts on this. The hypervisor in hypervisor setup >> with the ability to test multiple environments is also core to my thinking. >> I can't find any details on Trillian, other than what is in this thread. Is >> it open source and freely available? This is being developed at ShapeBlue? >> >> re:Bharat's Details: >> This seems like it is the most likely candidate for being able to offer >> value in the short term. Thank you for the work documenting it and making >> the source available. I still need to review the code. This approach will >> take quite a bit of work get setup initially, but once setup, it looks like >> it will do the trick. By using only freely available software makes it more >> accessible than some options which have been suggested in the past. >> >> My thoughts on the topic: >> - I think it is important that the CI can test all the different >> hypervisors and any hardware integrations available in the environment. >> - Ideally the CI tool would be easily distributed and installed at >> multiple sites. Every company who depends on ACS will have their own >> configuration, hardware and mission critical features. If the CI can be >> distributed (as independent installs), it will reduce the load on any >> specific site and it will enable companies who have specific hardware in >> their direct path to success the ability to validate their use case. >> - I think a standardized CI output is also important. Ideally the summary >> of the CI run would be posted back to the PR thread. I think it is also >> important to push the detailed logs to a central location so others can >> review them. Ideally, links to the detailed logs would also be pushed to >> the PR thread. >> - I think there should be multiple tiers of testing. Basic simulator >> testing and full runs. We may be able to setup simulator testing using a >> Docker container. I think @pdion891 has put some work into this that I may >> be able to build on. >> - I think it is probably too ambitious to do CI on every PR before merge, >> but ideally we would We may have to only CI the master branch for now and >> revert any PR that causes it to fail. >> >> I have been digging into this quite a bit today. I will continue working >> on this as it is very important to me for the 4.9 release. I would like to >> setup a CI environment locally as well to help validate and contribute to >> the effort. I have some hardware I can throw at this, so I will be working >> to get something setup to start testing ASAP. >> >> If I can work off either (or both) of your work, I would appreciate it. >> >> Cheers, >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw >> @CloudOps_ >> >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bharat.ku...@citrix.com> >> wrote: >> >> > Hi Paul, >> > >> > The goals and functionality of the CI on which we are working are the >> > same (please review the FS< >> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches >> > trator+and+test+runner+to+enable+continuos+integration> >> > ), The difference being the method of implementation. We are not using >> > nested virtualisation to create test environments. The work on CI >> > began long back but unfortunately due to other priorities we could not >> > work on a continuous basis and bring it to completion. >> > >> > I agree that we should come to a consensus on what the community wants >> > from the CI. IMHO the basic and immediate requirement would be to have >> > a stable system to test the PRs and post results. Once we have this, >> > we can talk about providing test setups to the community and next >> > steps. I think the availability of hardware has been a big hurdle >> > (even if we use virtualisation). >> > >> > Instead of working independently on the same thing let us work on this >> > together. I will share what we have currently, we can use it, or parts >> > of it (some of it is there at >> > https://github.com/bvbharatk/cloudstack-automation, but it is not >> > organised as in cannot be run out of the box). While virtualisation is >> > good from the ease of orchestration and resource utilisation point of >> > view, it has other problems like compatibility/dependencies issues. >> > Also it might tie the CI to a particular Hypervisor like ESXi as you >> have pointed out. >> > >> > So let us discuss on how to proceed further, and get things rolling. >> > >> > Thanks, >> > Bharat. >> > >> > >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.an...@shapeblue.com<mailto: >> > paul.an...@shapeblue.com>> wrote: >> > >> > Hi Bharat, >> > >> > Early last week I opened a discussion thread around the aims of >> > Trillian which we had begun working on as a CI/CD platform for the >> > community hardware (as well as individual use). Please could you >> > elaborate on where there is/isn't cross-over? In terms of >> > functionality and goals rather than method. It would be great to get >> > community consensus around what we want from our centralised CI/CD. >> > >> > >> > Project: Trillian >> > We have been working on PoC of a CI environment design which will: >> > * Provide fast build or rebuild of environments for testing. >> > * Enable multiple independent concurrent builds >> > * Be available on-demand through automation or individual request. >> > * Be capable of fully utilising all available hardware >> > * Flexible enough to be used to build super-realistic development >> > environments. >> > We intend to contribute and maintain our work within the Apache repos. >> > However, we are currently building the POC, figuring out the >> > requirements (and quirks) of the individual pieces, before pushing >> > something concrete for to the community to review. >> > >> > We envision that Trillian would cater for a number of use cases: >> > 1. CloudStack community integration testing of master against multiple >> > deployment scenarios (using ASF infra) 2. CloudStack community >> > integration testing of PRs against multiple deployment scenarios >> > (using ASF infra) 3. Organisations/individuals running the full suites >> > of tests available in Marvin against any physical environment they >> > have. >> > 4. Organisations/individuals deploying and running the full suites of >> > tests available in Marvin against virtualised infrastructures which >> > can be deployed by Marvin. >> > As we intend Trillian to test multiple environments concurrently, we >> > use nested virtualization on ESXi hosts (our testing has shown that >> > this is the only hypervisor which can support the nested >> > virtualisation of all other hypervisors with reasonable performance). >> > We use Ansible to deploy and configure all aspects of the build as >> > this will greatly lower the barrier to entry for independent testers. >> > We use CloudStack to provision the management server and virtualised >> > (nested) hosts on the physical hosts. We are creating Ansible >> > playbooks and roles which can: >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts >> > (inc. installing the relevant CloudStack agent where >> > required) >> > 3. Install required ACS packages on management server 4. Configure a >> > zone (including adding the compute hosts) via Marvin. >> > 5. Run the required Marvin tests. >> > 6. Return the results >> > We may need to propose enhancements to Marvin in order to sync the >> > configuration of hosts with the configuration used by Marvin. >> > >> > Using virtualised test environments, we can have multiple test >> > scenarios running concurrently. To do this we have found that it is >> > necessary to create pools or ranges of VLANs and IP addresses and >> > allocate them to environments. So for any given physical environment >> > which will be used for testing in, we take the total range(s) of IPs >> > and VLANs available and carve them into non-overlapping chunks >> > suitable for concurrent use as mgmt, public and guest networks. These >> > are stored in a MariaDB database. When a range is being used in a >> > testing environment, that range is marked as 'inuse' in the database. >> > When creating a test environment, Trillian looks in the database for >> > the next available VLAN range, the next available public IP range and >> > so on. The returned values are used to populate a Marvin cfg file >> > which in turn will be used to both build the environment and when >> > running the Marvin testing. When the virtualised infra is cleaned up, >> > the database will be updated to reflect that the used ranges are >> available again. >> > This initiative has only recently been started, and as stated earlier >> > we are currently figuring out the requirements (and quirks) of the >> > individual pieces and looking for the most suitable wrapper to glue it >> all together. >> > Also I have found that Marvin requires a little work to make the >> > output more meaningful/readable (especially in the case of errors and >> > exceptions) and to make it a little more intelligent about the tests >> > it can/can't run based on the chosen infrastructure components. I have >> > also found unreachable or very slow ISO and template paths hardcoded >> > into Marvin or individual tests. >> > We plan to enhance tests to address these issues and also reduce >> > runtimes where possible. >> > >> > >> > <http://www.shapeblue.com/> >> > Paul Angus >> > VP Technology , ShapeBlue >> > >> > >> > d: +44 203 617 0528 | s: +44 203 603 0540 >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> >> > | m: +44 7711 418784<tel:+44%207711%20418784> >> > >> > e: paul.an...@shapeblue.com | t: @cloudyangus<mailto: >> > paul.an...@shapeblue.com%20|%20t:%20@cloudyangus> | w: >> > www.shapeblue.com<http://www.shapeblue.com/> >> > >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK >> > >> > >> > >> > >> > >> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue >> > Services India LLP is a company incorporated in India and is operated >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda >> > is a company incorporated in Brasil and is operated under license from >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The >> > Republic of South Africa and is traded under license from Shape Blue >> > Ltd. ShapeBlue is a registered trademark. >> > This email and any attachments to it may be confidential and are >> > intended solely for the use of the individual to whom it is addressed. >> > Any views or opinions expressed are solely those of the author and do >> > not necessarily represent those of Shape Blue Ltd or related >> > companies. If you are not the intended recipient of this email, you >> > must neither take any action based upon its contents, nor copy or show >> > it to anyone. Please contact the sender if you believe you have received >> this email in error. >> > >> > >> > >> > >> > -----Original Message----- >> > From: Bharat Kumar [mailto:bharat.ku...@citrix.com] >> > Sent: Thursday, February 11, 2016 9:57 AM >> > To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> >> > Subject: Re: Important Pending Items >> > >> > Hi Sebastien, >> > >> > As Raja said, we are actively working on it and we will share the code >> > on github soon. >> > >> > Thanks, >> > Bharat. >> > >> > >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pull...@citrix.com >> > <mailto:raja.pull...@citrix.com>> wrote: >> > > >> > > Hi Sebastien, >> > > >> > > On item (3) - the BVTs have been running consistently (with same >> > passrates) and haven't had the time to automate the process of >> > reporting to an external site. >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression, >> > > working >> > on getting it run automatically and consistently. >> > > Hope to see it ready soon. >> > > >> > > Best, >> > > Raja >> > > -----Original Message----- >> > > From: Sebastien Goasguen [mailto:run...@gmail.com] >> > > Sent: Wednesday, February 10, 2016 2:29 PM >> > > To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> >> > > Subject: Important Pending Items >> > > >> > > Morning folks, >> > > >> > > We have several crucial pending items, that we need to resolve >> > > before >> > moving on: >> > > >> > > 1- We need an RM for master ( just saw some commits there that >> > > should be >> > reverted or merged properly in other branches). >> > > >> > > 2- We need to automate writing release notes, pushing/tagging new >> > > docs >> > when release come out and announcing releases on website. Currently >> > neither >> > 4.7 nor 4.8 have been announced. >> > > >> > > 3- CI is still almost inexistent >> > > >> > > -Sebastien >> > >> > Find out more about ShapeBlue and our range of CloudStack related >> services: >> > IaaS Cloud Design & Build< >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid >> > IaaS deployment framework<http://shapeblue.com/csforge/> >> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | >> > CloudStack Software Engineering< >> > http://shapeblue.com/cloudstack-software-engineering/> >> > CloudStack Infrastructure Support< >> > http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack >> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> >> > >> > >> Find out more about ShapeBlue and our range of CloudStack related services: >> IaaS Cloud Design & Build< >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid >> IaaS deployment framework<http://shapeblue.com/csforge/> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | >> CloudStack Software Engineering< >> http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure Support< >> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> >>
