[OpenStack-Infra] Adding index and views/dashboards for Kata to ELK stack
Hi Infra team. First, a brief overview for folks who have not been in this loop. Kata Containers has a CI that runs some metrics, and spits out JSON results. We'd like to store those results in the OSF logstash ELK infra (http://logstash.openstack.org/#/dashboard/file/logstash.json), and set up some Kibana views and dashboards so we can monitor historical trends and project health (and longer term maybe some advanced trend regression triggers). There is a relevant github Issue/thread showing a working PoC here: https://github.com/kata-containers/ci/issues/60#issuecomment-426579084 I believe we are at the point where we should set up a trial index and keys (for the filebeat/logstash/elastic injection) etc. so we can start to tie the parts together. What do I need to make that next step happen? Do I need to open a formal request ticket/Issue somewhere, or can we just thrash it out here? This email is part aimed at ClarkB :-), but obviously may involve others etc. Input most welcome. Thanks! Graham - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Adding index and views/dashboards for Kata to ELK stack
Hi Infra folks, et. al. Maybe this fell through the cracks due to conferences, travel etc., or maybe it was too broad spectrum... /me picks a pinpoint person or two... Thierry, Clark - is this something you can help with or facilitate a discussion around? Graham > -Original Message- > From: Whaley, Graham > Sent: Wednesday, October 3, 2018 11:17 AM > To: openstack-infra@lists.openstack.org > Cc: Anne Bertucio ; Ernst, Eric > Subject: Adding index and views/dashboards for Kata to ELK stack > > Hi Infra team. > > First, a brief overview for folks who have not been in this loop. > Kata Containers has a CI that runs some metrics, and spits out JSON results. > We'd like to store those results in the OSF logstash ELK infra > (http://logstash.openstack.org/#/dashboard/file/logstash.json), and set up > some Kibana views and dashboards so we can monitor historical trends and > project health (and longer term maybe some advanced trend regression > triggers). > > There is a relevant github Issue/thread showing a working PoC here: > https://github.com/kata-containers/ci/issues/60#issuecomment-426579084 > > I believe we are at the point where we should set up a trial index and keys > (for > the filebeat/logstash/elastic injection) etc. so we can start to tie the parts > together. > What do I need to make that next step happen? Do I need to open a formal > request ticket/Issue somewhere, or can we just thrash it out here? > > This email is part aimed at ClarkB :-), but obviously may involve others etc. > Input > most welcome. > > Thanks! > > Graham - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Adding index and views/dashboards for Kata to ELK stack
> Paul [0] and I [1] had both responded a while back with some thoughts on next > steps to make this possible, but those only went to the infra mailing list (I > didn't > check if you were subscribed). Hopefully the steps there make sense and we can > move forward with something along those lines? Ah, mea culpa! - I should have made it clearer I was not subscribed to the list, and pushed myself onto the CC. Apologies for that. Let's tackle what looks like the easier one first: > --- copied from the list --- > Yah, i think it would be great is we could use the JJB / grafyaml > workflow here where we store the view / dashboard in yaml then write a > job to push them into the application. Then we don't need to deal with > public authentication. If I understand then, we'd set up a kata git repo (or somewhere in our existing kata CI repo) to store the grafyaml files that define our views, and upon merges, a JJB defined Zuul job would take those, convert them to JSON and inject them into the grafana index of the elastic DB - that sounds nice to me, and I think should be workable. I like that flow. For the data injection itself then: > --- copied from the list --- > Currently all of our jobs submit jobs to the logstash processing system via > the ansible role here [0]. That is then fed through our logstash > configuration [1]. The first step > in this is probably to update the > logstash configuration and update the existing Kata jobs in zuul to submit > jobs to index that information as appropriate? Right, I think the logstash config will need updating to understand there is metrics JSON being injected, and to pass it on. We'll have to see how we get the filter for the JSON into that setup, and if we need (or can) have the filebeat inject a specific tag for instance. I don't think the Zuul Ansible role will be applicable - the metrics run on bare metal machines running Jenkins, and export their JSON results via a filebeat socket. My theory was we'd then add the socket input to the logstash server to receive from that filebeat - as in my gist at https://gist.github.com/grahamwhaley/aa730e6bbd6a8ceab82129042b186467 One crux here is that the metrics have to run on a machine with guaranteed performance (so not a shared/virtual cloud instance), and hence currently run under Jenkins and not on the OSF/Zuul CI infra. Let me know you see any issues with that Jenkins/filebeat/socket/JSON flow. I need to deploy a new machine to process master branch merges to generate the data (currently we have a machine that is processing PRs at submission, not merge, which is not the data we want to track long term). I'll let you know when I have that up and running. If we wanted to move on this earlier, then I could inject data to a test index from my local test setup - all it would need I believe is the valid keys for the filebeat->logstash connection. > Clark Thanks! Graham (now on copy ;-) - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Adding index and views/dashboards for Kata to ELK stack
(back to an old thread... this has rippled near the top of my pile again) > -Original Message- > From: Clark Boylan [mailto:cboy...@sapwetik.org] > Sent: Tuesday, October 23, 2018 6:03 PM > To: Whaley, Graham ; openstack- > in...@lists.openstack.org; thie...@openstack.org > Cc: Ernst, Eric ; fu...@yuggoth.org > Subject: Re: Adding index and views/dashboards for Kata to ELK stack [snip] > > I don't think the Zuul Ansible role will be applicable - the metrics run > > on bare metal machines running Jenkins, and export their JSON results > > via a filebeat socket. My theory was we'd then add the socket input to > > the logstash server to receive from that filebeat - as in my gist at > > > https://gist.github.com/grahamwhaley/aa730e6bbd6a8ceab82129042b186467 > > I don't think we would want to expose write access to the unauthenticated > logstash and elasticsearch system to external systems. The thing that makes > this > secure today is we (community infrastructure team) control the existing > writers. > The existing writers are available for your use (see below) should you decide > to > use them. My theory was we'd secure the connection at least using the logstash/beat SSL connection, and only we/the infra group would have access to the keys: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html The machines themselves are only accessible by the CNCF CIL owners and nominated Kata engineers with the keys. > > > > > One crux here is that the metrics have to run on a machine with > > guaranteed performance (so not a shared/virtual cloud instance), and > > hence currently run under Jenkins and not on the OSF/Zuul CI infra. > > Zuul (by way of Nodepool) can speak to arbitrary machines as long as they > speak > an ansible connection protocol. In this case the default of ssh would probably > work when tied to nodepool's static instance driver. The community > infrastructure happens to only talk to cloud VMs today because that is what we > have been given access to, but should be able to talk to other resources if > people show up with them. If we ignore the fact that all current Kata CI is running on Jenkins, and we are not presently transitioning to Zuul afaik, then Even if we did integrate the bare metal CNCF CIL packet.net machines vi ansible/SSH/nodepool/Zuul, then afaict you'd still be running the same CI tasks on the same machines and injecting the Elastic data through the same SSL socket/tunnel into Elastic. I know you'd like to keep as much of the infra under your control, but the only bit I think that would be different is the Jenkins Master. Given the Jenkins job running the slave only executes master branch merges, which have undergone peer review (which would be the same jobs that Zuul would run), then I'm not sure there is any security difference here in reality between having the Kata Jenkins master or Zuul drive the slaves. > > > > > Let me know you see any issues with that Jenkins/filebeat/socket/JSON flow. > > > > I need to deploy a new machine to process master branch merges to > > generate the data (currently we have a machine that is processing PRs at > > submission, not merge, which is not the data we want to track long > > term). I'll let you know when I have that up and running. If we wanted > > to move on this earlier, then I could inject data to a test index from > > my local test setup - all it would need I believe is the valid keys for > > the filebeat->logstash connection. Oh, I've deployed a Jenkins slave and job to test out the first stage of the flow btw: http://jenkins.katacontainers.io/job/kata-metrics-runtime-ubuntu-16-04-master/ > > > > > Clark > > Thanks! > > Graham (now on copy ;-) > > Ideally we'd make use of the existing community infrastructure as much as > possible to make this sustainable and secure. We are happy to modify our > existing tooling as necessary to do this. Update the logstash configuration, > add > Nodepool resources, have grafana talk to elasticsearch, and so on. I think the only key decision is if we can use the packet.net slaves as driven by the kata Jenkins master, or if we have to move the management of those into Zuul. For expediency and consistency with the rest of the Kata CI, obviously I lean heavily towards Jenkins. If we do have to go with Zuul, then I think we'll have to work out who has access to and how they can modify the Zuul job configs for Kata. (adding Salvador to CC, as he is the Kata Jenkins owner mostly, and has also worked on the Zuul PoC for Kata before). Graham (hoping we can come to some agreement :-) ) ---
Re: [OpenStack-Infra] Adding index and views/dashboards for Kata to ELK stack
Hi Clark, > There is more to it than that. This service is part of the CI system we > operate. > The way you consume it is through the use of Zuul jobs. If you want to inject > data into our Logstash/Elasticsearch system you do that by configuring your > jobs > in Zuul to do so. We are not in the business of operating one off solutions to > problems. We support a large variety of users and projects and using generic > flexible systems like this one is how we make that viable. > > Additionally these systems are community managed so that we can work > together to solve these problems in a way that gives the infra team > appropriate > administrative access while still allowing you and others to get specific work > done. Rather than avoid this tooling can we please attempt to use it when it > has > preexisting solutions to problems like this? We will happily do our best to > make > re-consumption of existing systems a success, but building one off solutions > to > solve problems that are already solved does not scale. > Sure, OK, understood... [snip] > > I wasn't directly involved with the decision making at the time but back at > the > beginning of the year my understanding was that Jenkins was chosen over Zuul > for expediency. This wasn't a bad choice as the Github support in Zuul was > still > quite new (though having more users would likely have pushed it along more > quickly). It probably would be worthwhile to decide separately if Jenkins is > the > permanent solution to the Kata CI tooling problem, or if we should continue to > push for Zuul. If we want to push for Zuul then I think we need to stop > choosing > Jenkins as a default and start implementing new stuff in Zuul then move the > existing CI as Kata is able. > > As for who has Zuul access, the Infra team has administrative access to the > service. Zuul configuration for the existing Kata jobs is done through a repo > managed by the infra team, but anyone can push and propose changes to this > repo. The reason for this is Zuul wants to gate its config updates to prevent > new > configs from being merged without being tested. Bypassing this testing does > allow you to break your Zuul configuration. Currently we aren't gating Kata > with > Zuul so the configs live in the Infra repo. If we started gating Kata changes > with > Zuul we could move the configs into Kata repos and Kata could self manage > them. > > Looking ahead Zuul is multitenant aware, and we could deploy a Kata tenant. > This would give Kata a bit more freedom to configure its Zuul pipeline > behavior > as desired, though gating is still strongly recommended as that will prevent > broken configs from merging. I spoke with some of the other Kata folks - we agreed I'd try to move the Kata metrics CI into Zuul utilizing the packet.net hardware, and let's see how that pans out. I think that will help both sides understand the current state of kata/zuul so we can move things forwards there. Wrt the packet.net slaves, I believe we can do that using some of the packet.net/zuul integration work done by JohnStudarus - John and I had some chats at the Summit in Berlin. https://opensource.com/article/18/10/building-zuul-cicd-cloud I'll do some Zuul readup, work out how I need to PR the additional ansible/yaml items to the infra repos to add the metrics build/runs (I see the repos and code, and a metrics run is very very similar to a normal kata CI run - and to begin with we can do those runs in the VM builders to test out the flows before moving to the packet.net hardware). [move this down to the end...] > > No, we would inject the data through the existing test node -> Zuul -> > Logstash - > > Elasticsearch path. This might be one bit we have to work out. The metrics generates raw JSON results. The best method I found for landing that directly into logstash was through the socket filebeat. It is not clear in my head how this ties in with Zuul - the best method I found previously for direct JSON injection into logstash, and thus elastic, was using the socket filebeat. Will that fit in with the infra? Graham - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Feedback on git/repos on opendev.org
Hi, (please reply-all if you want me to see it - I am not subscribed to this list) I was heading to read the code for the kata zuul tenant, and had some niggles when trying to locate/navigate. I thought I'd at least drop the feedback here (thanks ttx). Some of it will be specific to my 'flow' Sooo, as I don't store the path to the repos in my head, I normally start out by going to git.openstack.org: That used to get me to (iirc) the top level cgit type interface, where I could then navigate down to the kata repos. It now takes me to the top level of opendev.org, but not to the git area, and it is not immediately obvious to me where the git area is. I found the git area under 'Explore' :-). The 'search' is a bit odd. It took me a moment to work out that it is alphabetical, if you ignore all the subdir prefix's (project names?). I'm not sure that is the most useful sort order tbh? Confused me for some time. And then, if you search for 'kata' in the default search box, you get **no matching repos**. Which, also, is not quite what I'd expect. Eventually I found the kata repos by sorting reverse-alphabetically and scrolling down :-) I know, the search is on 'repos', and not 'organisations' - but, if you just land on that page and search for something that happens to only have its name active at the project level (like kata), then you end up with no results. Maybe the search/results can be smarter, and provide the list of what it was searching for first (so in this case 'no repositories found'), and then some 'suggestions', like 'you may have been looking for organisation: kata'. Just thinking out loud. Thanks, Graham - Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
