So another idea that was talked about on IRC. Taskflow exposes entrypoints for these storage backends (like your storage callback/interface idea).
It currently provides 3 such 'default' backends [sqlalchemy, file/dir based, in-memory <--> mainly for testing]. A 4th one is in progress for icehouse (zookeeper based). These backends are used to store intermediary 'flow/task' state (allowing the taskflow engine to resume if an app crashes/stopped while doing stuff). A couple ideas about this, since its already pluggable. Split the sqlalchemy backend -> 'taskflow-sa' repo/new package. For those projects that want to use this backend, they include it (still means 'taskflow-sa' package has a dependency on sqlalchemy of some version). Another idea is to move the sqlalchemy dependency currently in taskflow requirements.txt -> taskflow test-requirements.txt and for those projects that want to use the sqlalchemy backend, they include the sqlalchemy version themselves in there requirements.txt (taskflow keeps it in test-requirements.txt so that it can run its unit/integration tests, making sure the backend still works). I'm not really sure which is the best, none seem super-great, but sometimes u just work with what u got :-P On 1/3/14, 3:32 PM, "Sean Dague" <s...@dague.net> wrote: >On 01/03/2014 06:14 PM, Joshua Harlow wrote: >> Since the library model is what most everyone else uses outside of >> openstack (I assume?) what can we do to get there so that this model >>works >> as well? >> >> Expanding dependencies recursively seems like it could help? This could >> then detect transitive dependency issues (and doesn't seem so hard to >>do). > >We actually talked about having pip be able to help us here with a >--requirements-override piece of function. dstufft seemed positive on >the concept. > >> I like the idea of 'gate on all the things' (that I've heard come up >> before) but I don't know if its possible? If we hooked into the pypi >> upload 'stream' it would seem like we could automatically trigger >> openstack builds when a known dependency (or dependency of a >> dependency...) is uploaded (maybe?). That would be pretty neat. > > >> In general it really seems like having more libraries, not less is ideal >> (making us solve this issue whether we want to or not really). As >> libraries can then be used outside of openstack (taskflow is already >>being >> used elsewhere also), libraries create well-defined apis and boundaries >> between programs (...). I know they also create this dependency hell >> problem (anvil has been hitting these same issues for a while to). But I >> think we can figure out a solution that fits both worlds, the world of >> things we can gate on and the world of things we can't (3rd party >> libraries that aren't under openstack control). Taskflow is in-between >> those worlds, so it allows us to explore how to make both of those >>worlds >> work. > >In general I agree.... however, if you get between OpenStack and SQLA >you've now touched the 3rd rail. Because we have deep experience about >how bad the compatibility between versions can be, and we can't be >beholden to another project about our SQLA upgrade timeframe. > >So I think that generalities aside, if are a library, and use SQLA, we >probably need to really think about putting it in the integrated gate. > >Because otherwise what we are saying is taskflow is completely dictating >the SQLA version in the Icehouse release of OpenStack. And that's the >wrong place for that decision to be. > >If taskflow worked with a storage callback mechanism, and got a storage >interface from the program that was calling it, then things would be >much different. But because it's going straight to the database and >managing tables directly, through a known unstable library, that >OpenStack itself needs some control over, it's definitely a different >case. > > -Sean > >-- >Sean Dague >Samsung Research America >s...@dague.net / sean.da...@samsung.com >http://dague.net > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
