On 06/09/2014 12:11 AM, Jesus M. Gonzalez-Barahona wrote: > On Sun, 2014-06-08 at 13:32 -0400, Monty Taylor wrote: >> On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote: >>> On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote: >>>> Hello folks >>>> >>>> I wanted to get your thoughts on the idea to move the whole Bitergia >>>> grimoire engine, scripts, database, etc to the openstack common >>>> infrastructure. I'm sure everybody will want this so the question is >>>> more about resources as in people willing to help Bitergia get their >>>> machinery puppetized the "OpenStack-CI way". >>>> >>>> At the moment the various spiders and scripts run on a machine on >>>> Bitergia's end and they drop the results of the elaboration (json, html, >>>> css files and sql dumps) on activity.openstack.org/dash/. >>>> >>>> While this setup is convenient and has been working so far, I think >>>> we've outgrown it. One of the areas that we want to do more with is the >>>> datawarehouse built by Bitergia, enable it to serve other purposes too >>>> and allow more collaboration from the community. For example, once the >>>> tools are all out in one place, interested parties could build some sort >>>> of service on top of your datawarehouse to export the data about the >>>> affiliation. Others could build tools to 'fix' such affiliation, pulling >>>> the various mailmap files used by gitdm and stackalytics. >>>> >>>> The question for OpenStack-CI is then: in the next weeks/months, is >>>> there going to be someone free, from the CI team or someone willing to >>>> join it (Dan?), to help Bitergia's team get their tools on our >>>> infrastructure? >>> >>> Thanks for the introduction of the issue, Stefano. >>> >>> Just to clarify a bit the needs to move all the Grimoire machinery to a >>> vm under openstack-ci, the software is split in three main parts: >> >> Awesome. >> >>> * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.), >>> and store the data into a MySQL database (well, in fact, one schema per >>> kind of repository). >>> >>> * The MySQL database itself. >>> >>> * The vizGrimoire tools, that run the analysis, produce JSON files, and >>> the HTML/CSS/JavaScript files needed to serve the dashboard. All of this >>> is for now static, which means that you only need to serve those files >>> via HTTP, and you're done: no live queries to the database once the JSON >>> files are produced. >>> >>> Right now, we produce the JSON files once a day. That means that >>> MetricsGrimoire is first run once a day (the tools know how to get >>> incremental information from repositories), data is updated in the >>> database, and then vizGrimoire analysis is run. All of this is >>> controlled by automator, the tool that is configured with the list of >>> repos to analyze, the analysis to run, etc. >>> >>> MetricsGrimoire tools are written in Python. There are some Python >>> dependencies beyond Python 2.7, but I guess all of them are >>> straightforward from pypy. >> >> Where are these stored/installed from? Is it "pip install >> MetricsGrimoire"? Is it in a git repo? Does this git repo want to move >> into openstack tooling or does it want to stay where it is and have >> openstack-infra consume releases? > > Those are different tools in git repos. Providing them via pip could be > a part of the process, anyway: that's something we have in the roadmap > for a while. Right now, openstack-infra would consume the repositories.
git repos is fine - we install several things directly from git - just was curious. I'm assuming based on this email so far that you want to keep your git repos in github? > More info about MetricsGrimoire, and its repos: > > http://metricsgrimoire.github.io Sweet. Thanks. >>> The MySQL is a plain MySQL. It would be great having it in SSD, because >>> that speeds up queries a lot. But probably it will work with regular >>> disks. >> >> We use cloud-based databases from Rackspace's Trove service. I have no >> idea what sort of disk is behind it - but MySQL databases are no problem. > > ok. It's mainly a matter of performance, I guess Trove would be ok. > >>> Most of vizGrimoire is also Python, but there is still some R code >>> (we're currently moving towards a pure-Python implementation). Python >>> and R dependencies are easy too (from pypy or CRAN). >> >> Same questions as above on MetricsGrimoire. Do you have docs anywhere on >> installing these two things? > > For MetricsGrimoire, it is very close to "just standard Python install". > But vizGrimoire is another story. Right now, it is a moving target, so > probably the best thing would be that we install it, and document the > procedure in detail for you. In the process, we could automate some > steps that are not automated yet. Let's work through your script below and the puppet first. There's not much value in having you install something on a server because then we don't know anything about running it in the future, and if there is a major failure and we have to rebuild, we won't be confident that our automation can do the job. You have the code running currently and publishing to activity - so I don't think there is an urgency in getting the code running elsewhere- the real benefit is in getting it tied in to normal process for us so we can run it together. > In any case, some more info about vizGrimoire: > > http://vizgrimoire.github.io > > I guess the closer thing we have to a install doc is: > > https://github.com/MetricsGrimoire/Automator/wiki/Install Sweet. Putting it on my reading list. > (Automator is the script that runs all the party) > >>> We're usually deploying on Debian or Ubuntu, but we have some experience >>> with other Linux-based OSs too. >> >> Great. All of our services run on Ubuntu Precise right now. We'll be >> starting to think about an upgrade to trusty. >> >>> We could do the whole deployment, if you can provide us with a >>> Debian/Ubuntu vm, including proper documentation to reproduce it if >>> needed. >> >> All of our deployments are done in a fully automated manner driven from >> our puppet repo: https://git.openstack.org/cgit/openstack-infra/config. >> However - we'd be more than happy for you to write the puppet! (and >> happy to help with pointers) >> >> I just wrote the puppet to run stackalytics in Infra: >> >> https://review.openstack.org/98656 >> >> It's probably a great place to start in terms of looking at puppetizing >> activity board. If someone can write up something similar to: >> >> https://wiki.openstack.org/wiki/Stackalytics/HowToRun >> >> I'll be more likely to do what I did today which is wake up on a Sunday >> and lazily write some puppet while watching Rafael Nadal win his ninth >> French Open. :) > > ;-) > > Thanks for the pointers, Monty. Therefore, I assume that the first thing > to do is to have a detailed installation information and/or Puppet > config files. We're not that familiar with Puppet, so maybe the best > thing for us would be to start writing the detailed docs, and after that > starting with the Puppet thing... Right? Oh - you'll know puppet for sure by the time we get done with you. :) Seriously though - more than happy to help drive the puppet side of things if you have detailed instructions on how to install. I want you to learn and be involved in that - but I also don't want you to be totally blocked on the concept. _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
