Data processing pipeline workflow management

2015-03-10 Thread Lars Albertsson
r, counter) counter_pipe_backfill = Tasks.concat(*[counter_pipe_minutely.bind( hour_offset=Integer(offset)) for offset in range(5)]) job_template = Job( role='www-data', contact='lars.alberts...@gmail.com', instances=1, service=False, task=counter_pipe_backfill, **

Re: Data processing pipeline workflow management

2015-03-24 Thread Lars Albertsson
jobs when inputs are available, but output is missing. I could probably make it convenient for users by extending the Aurora DSL in order to keep all job specification information close. I'll let you know if I get around to it. Regards, Lars Albertsson On Wed, Mar 11, 2015 at 11:21 PM, Z