I am going to open a ticket with the infra team to request access tokens for each of the organizations who are putting up hardware for the CI cause.
The reason I am planning to request a token for each organization individually is because I want to make a point about our need for CI and the need for many groups to be doing CI in order for us to get full hardware coverage of the different features we support. The token is needed in order for the different CI implementations to be able to simply post back a 'red light', 'green light' status of the pull request in github. More details can be added in comments using an implementation like what Bharat is working on, but the focus of this request will be to enable us to have the ability for the different CI environments to get at least 'red light', 'green light' functionality for pull requests. I will be requesting a token with the following basic permissions: - public_repo - the same as an anonymous user - repo:status - this allows for the status of a PR to be changed to one of the following 4 states; pending, success, failure, error, with some additional information such as; context, description, and a public url for more details of the status. I am going to associate people with each token request and ask that each person be sent their own token since they are providing infrastructure to operate a CI environment and the community would like visibility into their results. The people I am going to ask for a token for are as follows: Will Stevens <wstev...@cloudops.com> Paul Angus <paul.an...@shapeblue.com> Bharat Kumar <bharat.ku...@accelerite.com> Remi Bergsma <rberg...@schubergphilis.com> If you would also like to be included in this request and get your own token in order to be able to contribute back the status of your CI runs, please let me know so I can request a token for you as well. Ilya, you had shown some interest in this as well, would you like me to include you? If so which email address should I use? Anyone else who is planning to contribute an environment to the CI process in the near to medium future? Cheers, *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Mon, Mar 7, 2016 at 6:36 PM, Erik Weber <terbol...@gmail.com> wrote: > I guess the appropriate channel would be to create a jira ticket for INFRA: > https://issues.apache.org/jira/browse/INFRA > > -- > Erik > > On Mon, Mar 7, 2016 at 11:18 PM, Will Stevens <wstev...@cloudops.com> > wrote: > > > This is kind of what I was expecting. Do you know who I would be > > contacting? The permissions required are VERY minimal AND they have > > already given the 'TravisCI' application the same permissions as we need > > for this. > > > > How did we get the TravisCI application enabled and the permissions > > accepted for that integration? > > > > I have been trying to thinking of ways to potentially work the system > from > > that side. The main issue I have with this approach is that it is not a > > single application that we want to give permission to. Ideally, we would > > give each individual/organization who is contributing a CI environment > > their own token. In that case, we would have to register the CI of each > > party as their own CI integration. > > > > Here are some ideas I have as a 'work around' to the problem: > > > > 1) Register a single upr CI application with Github and have the apache > > guys enable the integration. This will give the application a single > > access token. I can then compile upr with the access token embedded into > > the binary. I don't like this approach and I feel we would probably be > > violating some ToS somewhere. > > > > 2) Provide a web server implementation that each CI party can use to > > register their own CI endpoint as a Github application integration. Then > > we have the apache guys enable each of them (which will be a harder sell > to > > them), but then each CI party will get their own token and will be able > to > > post back as themselves. This is also nice because if someone is not a > > good community member and misbehaves, their integration can be revoked > > without it affecting everyone else who has a CI integration. > > > > 3) Provide a single web server implementation that is registered as a > > Github Application Integration. This implementation is then approved by > > the apache guys for the cloudstack repo. This web server implementation > > (let's call it upr_server) keeps our one and only access token. I then > > modify the upr command line tool to take a token that is provided by the > > upr_server when a CI party registers on the upr_server website. The > > upr command > > will actually target the upr_server box when posting statuses, etc, which > > will essentially proxy the calls on to Github using the token that was > > approved for the integration. > > > > I think that 3) is probably the cleanest solution and would reduce our > > chances of getting banned by someone for being too cheeky. It is a whole > > lot of trouble for nothing, but if they are going to be a stick in the > mud > > about this and not allow access tokens to anything other than official > > github supported integrations, then I will have to make that work... > > > > Ideas? Thoughts? Rants? :P > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > w cloudops.com *|* tw @CloudOps_ > > > > On Mon, Mar 7, 2016 at 4:12 PM, Remi Bergsma < > rberg...@schubergphilis.com> > > wrote: > > > > > Hi Will, > > > > > > This is the main problem: there’s no one except Apache Infra with > access > > > to the Github CloudStack repo. Even committers have to push to Apache > > git, > > > which is mirrored to Github. We can’t close a PR, set a label, change a > > > title or whatever basic operation. You can ask them for a token. When I > > (as > > > the release manager) asked for any more permission than an anonymous > user > > > has it was kindly refused. I really hope you’ll have more luck but I > > > wouldn’t count on it. > > > > > > > > > > > > Regards, > > > Remi > > > > > > > > > On 07/03/16 19:10, "williamstev...@gmail.com on behalf of Will > Stevens" > > < > > > williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote: > > > > > > >The main thing we have to sort out with this type of integration (as > it > > is > > > >today) is the distribution of access tokens with the correct > permissions > > > on > > > >the apache/cloudstack github repo. The required permissions are very > > > >limited, but I don't know if we have access to create new tokens. If > we > > > >don't then I will have to develop an application integration > workaround > > to > > > >make it easier for the people with access to the apache/cloudstack > repo > > to > > > >give the people running CI integrations access to update statuses > (like > > > the > > > >current travis integration). > > > > > > > > > > > > > > > > > > >If you have questions or feedback, please don't be shy. > > > > > > > >*Will STEVENS* > > > >Lead Developer > > > > > > > >*CloudOps* *| *Cloud Solutions Experts > > > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > > >w cloudops.com *|* tw @CloudOps_ > > > > > > > >On Mon, Mar 7, 2016 at 12:36 PM, Bharat Kumar < > > > bharat.ku...@accelerite.com> > > > >wrote: > > > > > > > >> Hi guys, > > > >> > > > >> I am also working on the similar reporting problem, here is what i > did > > > >> > > > >> link to the report https://github.com/bvbharatk/cloud-stack/pull/1 > > > >> > > > >> I am thinking this is good enough for now, I want to start posting > the > > > >> results on each pr as shown in the above link. > > > >> please give me your comments or suggestions. > > > >> > > > >> Thanks, > > > >> Bharat > > > >> On 05-Mar-2016, at 7:02 PM, Will Stevens <wstev...@cloudops.com > > <mailto: > > > >> wstev...@cloudops.com>> wrote: > > > >> > > > >> Daan > > > >> > > > >> Regarding the obligatory provider id. I agree, but I am still > trying > > to > > > >> figure out the details. Creating distinct runs that have their own > > > status > > > >> is done by setting the 'context'. I think we would need to have two > > > pieces > > > >> to this. A provider id and an environment id. > > > >> > > > >> So for example. Lets assume that my provider id is 'CloudOps' and I > > > have > > > >> two different environments, one for 'KVM' and one for 'Xen' (for > > > example). > > > >> I would then the tool would produce the following two independent CI > > > >> statuses. > > > >> 'CloudOps - KVM' : with a basic description of the environment. > > > >> 'CloudOps - Xen' : again with a basic description of the env. > > > >> ... and so on ... > > > >> > > > >> I am still sorting out the details here as well as making it easy > for > > > us to > > > >> integrate this into the apache/cloudstack repo with the access we > > > currently > > > >> have. Adding this is 'no biggy' for me because I am building this > > tool > > > as > > > >> we speak, and trying to tailor it to our (ACS) needs, so this type > of > > > >> feedback is perfect as it allows me to adapt the tool before I get > too > > > >> deep. > > > >> > > > >> *Will STEVENS* > > > >> Lead Developer > > > >> > > > >> *CloudOps* *| *Cloud Solutions Experts > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_ > > > >> > > > >> On Sat, Mar 5, 2016 at 4:17 AM, Daan Hoogland < > > daan.hoogl...@gmail.com > > > >> <mailto:daan.hoogl...@gmail.com>> > > > >> wrote: > > > >> > > > >> Will > > > >> > > > >> Gret work, especially the thing you are showing in link [4], I would > > > like > > > >> to make an enhancement request and that is a obligatory provider id. > > > Only > > > >> if it is no biggy for you! > > > >> Several people may decide to do a XVM on ChildrensOS for instance > and > > > so we > > > >> may be aware of an obscurity that is different. If one fails and the > > > other > > > >> succeeds it is easily identified. > > > >> > > > >> Ilya, > > > >> > > > >> I have been playing with go and it is a very nice language for such > a > > > >> simple script, though it wasn't exactly designed for it. So don't > read > > > my > > > >> comment/question as an objection. But we do have > > > >> bash,python,c#,java,javascript,xslt,sql at least. That is not > counting > > > the > > > >> build system and I am sure the hyperv has some extra windows > specific > > > >> stuff. > > > >> To me it is inherent to the nature of across platform orchestration > > and > > > >> provisioning system so it is fine. It is something to consider. On > the > > > >> other hand when bringing in new tools we don't make the choice > so.... > > I > > > am > > > >> ranting, I guess. > > > >> > > > >> > > > >> > > > >> On Sat, Mar 5, 2016 at 7:38 AM, ilya <ilya.mailing.li...@gmail.com > > > <mailto: > > > >> ilya.mailing.li...@gmail.com>> wrote: > > > >> > > > >> I see where Daan is coming from :) I thought this would be 4th, not > > > >> exactly 7ths. > > > >> > > > >> I'm not against golang by any means (if anything - its my next "go" > to > > > >> language these days). > > > >> > > > >> Things to consider: > > > >> > > > >> Would notify-pr support proxy? I've been thinking on ways of > > > >> contributing test runs, there would have to be few things i'd need > to > > > do. > > > >> > > > >> 1) massage the log content - such that no environment details are > > > >> exposed, i can probably handle this with sed/awk.. > > > >> > > > >> 2) i'm behind multiple firewalls with no internet access. however, > > some > > > >> lab environments might have a proxy, so it would be nice to have a > > > >> support for it. > > > >> > > > >> Thanks > > > >> ilya > > > >> > > > >> > > > >> > > > >> On 3/4/16 6:56 AM, Will Stevens wrote: > > > >> Yes, I have most of it already built and will be releasing it later > > > >> today > > > >> or over the weekend. The reason I chose Golang is because it can be > > > >> cross > > > >> compiled to be run on any system and distributed as a single binary > > > >> with > > > >> no > > > >> dependencies. This means that no one will have to worry about > > building > > > >> it > > > >> or having to change their environment at all in order to use it. I > am > > > >> trying to lower the barrier to entry and make it as easy as possible > > > >> for > > > >> people to contribute back their CI execution details. > > > >> > > > >> *Will STEVENS* > > > >> Lead Developer > > > >> > > > >> *CloudOps* *| *Cloud Solutions Experts > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_ > > > >> > > > >> On Fri, Mar 4, 2016 at 8:53 AM, Daan Hoogland < > > daan.hoogl...@gmail.com > > > >> <mailto:daan.hoogl...@gmail.com> > > > >> > > > >> wrote: > > > >> > > > >> Will, Do you have an implementation of notify-pr? I am asking as you > > > >> specify it will be implemented in golang which seems odd. It is not > > > >> amongst > > > >> the 7 or so languages already in use. > > > >> > > > >> On Fri, Mar 4, 2016 at 1:54 AM, Will Stevens < > > > >> williamstev...@gmail.com<mailto:williamstev...@gmail.com>> > > > >> wrote: > > > >> > > > >> Hey Everyone, > > > >> As I am sure most of you are aware, I have been focusing a lot on > > > >> ways > > > >> to > > > >> get CI integrated back into the community. > > > >> > > > >> Today I build a little POC to validate some ideas and get a feel for > > > >> a > > > >> potential approach for getting CI integrated into the Github pull > > > >> request > > > >> workflow. > > > >> > > > >> There are multiple individuals/companies focusing on CI right now > > > >> (which > > > >> is a good thing), but there has not really been much discussion > > > >> (that I > > > >> am > > > >> aware of) for how these different CI runs will push back results to > > > >> the > > > >> community. I want to make sure that nobody's work on this topic > goes > > > >> to > > > >> waste, so my goal is to provide a simple and consistent way for > > > >> everyone > > > >> to > > > >> push their results back to the community. > > > >> > > > >> Here is the basic idea (please give feedback): > > > >> - A simple cross platform command line tool with zero dependencies > > > >> will > > > >> be > > > >> provided (and open sourced) which will handle the submission of the > > > >> CI > > > >> results back to the community. It is written in Golang and is > > > >> currently > > > >> called 'notify_pr'. > > > >> - At the end of a CI execution, the CI should automate the execution > > > >> of > > > >> this tool to handle updating the appropriate PR with the results. > > > >> > > > >> Configuration can be done via the command line or through an INI > > > >> file. > > > >> Here is an example of the configuration details. The commit is the > > > >> commit that the CI just executed against. > > > >> > > > >> token = <your github token> > > > >> owner = apache > > > >> repo = cloudstack > > > >> commit = c8443496d3fad78a4b1a48a0992ce545bde299e8 > > > >> > > > >> summary_file = <a text file summary of the run> > > > >> full_detail_dir = <a directory structure to be recursively uploaded > > > >> to > > > >> object store> > > > >> full_detail_files = <a comma separated list of files to upload to > > > >> object > > > >> store> > > > >> store_api = <swift or s3> > > > >> store_endpoint = <url endpoint> > > > >> store_identity = <keystone identity or aws access key> > > > >> store_secret = <keystone password or aws secret key> > > > >> > > > >> I have not yet implemented the object storage endpoints, but I have > > > >> code > > > >> to do it from a different project, so I just need to add it. I will > > > >> be > > > >> able to host my CI output in a swift object store, but others may > > > >> need > > > >> to > > > >> use AWS S3. Maybe we can get sponsorship for this storage. We will > > > >> only > > > >> keep the logs for a window of like a week or so on the object store > > > >> so > > > >> the > > > >> storage usage will not be ever growing. > > > >> > > > >> Basically, the tool takes the details of the repository you are > > > >> validating > > > >> a Pull Request for and the commit you are building. It also takes > > > >> the > > > >> files you would like to push to the pull request. The summary file > > > >> will > > > >> be > > > >> shown as text in the pull request comment and the other files will > be > > > >> uploaded to an object store and will be publically available for a > > > >> period > > > >> of time (probably about a week and then get cleaned up, details > TBD). > > > >> The > > > >> files to be uploaded to object store could be either specified as > > > >> individual files, OR a target directory and all the files in that > > > >> directory > > > >> will be recursively uploaded to the object store. > > > >> > > > >> When the tool is run, it will scan through all the open pull > requests > > > >> in > > > >> the target repository and when it finds the pull request > > > >> corresponding > > > >> to > > > >> the commit in question, it will post the details as a comment to > that > > > >> pull > > > >> request. This functionality is currently working (see the attached > > > >> screenshot). I can change the formatting and such, this is just an > > > >> example. > > > >> > > > >> This is still a very rough concept that I have only worked on for a > > > >> day, > > > >> but hopefully you guys agree that it is a good start towards a > > > >> consistent > > > >> feedback mechanism for the community to take advantage of the > > > >> different > > > >> distributed CI installations. > > > >> > > > >> Please voice your feedback and concerns. I am sure I have not > > > >> thought > > > >> of > > > >> everything and we may still want to make fundamental changes to the > > > >> approach, but I think the concept is solid. > > > >> > > > >> Cheers, > > > >> > > > >> Will > > > >> > > > >> > > > >> > > > >> > > > >> -- > > > >> Daan > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> -- > > > >> Daan > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> DISCLAIMER > > > >> ========== > > > >> This e-mail may contain privileged and confidential information > which > > is > > > >> the property of Accelerite, a Persistent Systems business. It is > > > intended > > > >> only for the use of the individual or entity to which it is > addressed. > > > If > > > >> you are not the intended recipient, you are not authorized to read, > > > retain, > > > >> copy, print, distribute or use this message. If you have received > this > > > >> communication in error, please notify the sender and delete all > copies > > > of > > > >> this message. Accelerite, a Persistent Systems business does not > > accept > > > any > > > >> liability for virus infected mails. > > > >> > > > > > >
