Re: Data processing pipeline workflow management

2015-03-24 Thread Lars Albertsson
Thanks for good advice and input everyone! While we were talking, Pinterest open sourced their workflow engine: http://engineering.pinterest.com/post/113376157699/open-sourcing-pinball. It looks similar to Luigi in terms of architecture. My current plan is to use Aurora in the manner described in

Graceful task shutdown

2015-03-24 Thread Erb, Stephan
Hi everyone, we are implementing the /health endpoint in our services but omit the implementation of the unauthenticated lifecycle methods /quitquitquit and /abortabortabort. As a consequence, stopping a service is taxed by 10 seconds waiting time [1]. I would like to get rid of this unnecess

Re: Graceful task shutdown

2015-03-24 Thread Brian Brazil
On 24 March 2015 at 20:57, Erb, Stephan wrote: > Hi everyone, > > we are implementing the /health endpoint in our services but omit the > implementation of the unauthenticated lifecycle methods /quitquitquit and > /abortabortabort. > > As a consequence, stopping a service is taxed by 10 seconds w

Re: Graceful task shutdown

2015-03-24 Thread George Sirois
Unfortunately I don't think my change will be able to make it in as-is. As Brian Wickman pointed out, it could introduce serious problems because there are varying timeouts across the scheduler/executor, so if you set your wait time to be too high, the scheduler might start to consider the tasks l

Re: Graceful task shutdown

2015-03-24 Thread Brian Brazil
On 24 March 2015 at 21:33, George Sirois wrote: > Unfortunately I don't think my change will be able to make it in as-is. > > As Brian Wickman pointed out, it could introduce serious problems because > there are varying timeouts across the scheduler/executor, so if you set > your wait time to be