On 09/05/2014 04:21 PM, Monty Taylor wrote: > On 09/05/2014 06:32 AM, Sean Dague wrote: >> While reviewing this zookeeper service group fix in Nova - >> https://review.openstack.org/#/c/102639/ it was exposed that the >> zookeeper tests aren't running in infra. >> >> The crux of the issue is that zookeeper python modules are C extensions. >> So you have to either install from packages (which we don't do in unit >> tests) or install from pip, which means forcing zookeeper dev packages >> locally. Realistically this is the same issue we end up with for mysql >> and pg, but given their wider usage we just forced that pain on >> developers. >> >> But it seems like a bad stand off between testing upstream and testing >> normal path locally. >> >> Big picture it would be nice to not require a ton of dev libraries >> locally for optional components, but still test them upstream. So that >> in the base case I'm not running zookeeper locally, but if it fails >> upstream because I broke something in zookeeper, it's easy enough to >> spin up that dev env that has it. >> >> Which feels like we need some decoupling on our requirements vs. tox >> targets to get there. CC to Monty and Clark as our super awesome tox >> hackers to help figure out if there is a path forward here that makes >> sense. > > Funny story - I've come to dislike what we're doing here, so I've been > slowly working on an idea in this area: > > https://github.com/emonty/dox > > The tl;dr is "it's like tox, except it uses docker instead of > virtualenv" - which means we can express all of our requirements, not > just pip ones. > > It's not quite ready yet - although I'd be happy to move it in to > stackforge or even openstack-dev and get other people hacking on it with > me until it is. The main problem that needs solving, I think, is how to > sanely express multiple target environments (like py26,py27) without > making a stupidly baroque config file. OTOH, tox's insistence of making > a new virtualenv for each "environment" is heavyweight and has led to > some pretty crazy hacks across the project. Luckily, docker itself does > an EXCELLENT job at handling caching and reuse - so I think we can have > a set of containers that something in infra (waves hands) publishes to > dockerhub, like: > > infra/py27 > infra/py26 > > And then have things like nova build on those, like: > > infra/nova/py27 > > Which would have zookeeper as well > > The _really_ fun part, again, if we can figure out how to express it in > config without reimplementing make accidentally, is that we could start > to have things like: > > infra/mysql > infra/postgres > infra/mongodb > > And have dox files say things like: > > Nova unittests want a python27 environment, this means we want an > infra/mysql container, an infra/postgres container and for those to be > linked to the infra/nova/py27 container where the tests will run. > > Since those are all reusable, the speed should be _Excellent_ and we > should be able to more easily get more things runnable locally without > full devstack. > > Thoughts? Anybody wanna hack on it with me? I think it could wind up > being a pretty useful tool for folks outside of OpenStack too if we get > it right. >
I think it is sexy - I don't describe ideas/software as sexy that often but this one deserves it. I'm interested in helping out. I'll clone it and give it a try - or at least take a look at it. Flavio > Monty > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
