Try GNUbatch.
On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak <w...@zak.co.il> wrote:
>For dependency management, you may want to use 'make' or modern
>equivalents ('ant', 'gradle', etc.).
>For controlling remote nodes, 'ansible' may be able to do the work.
>
>--- Omer Zak
>
>
>On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
>> Hi all,
>>
>> I need some advice, currently I have a huge cron file which schedules
>> tasks one after anther, and each task is position precisely (with
>> some room for error) to start after it predecessor.
>>
>> So if one job start at 00:00 and it will go and fetch some files and
>> it takes 3minutes
>> the next job will be after start right after at ~00:05
>> and so on
>>
>> the problem is that if one job fails, all other jobs which are depend
>> on him will fail as well, and then I get a shitload of alerts, and
>> the worst part is that if i have to manually start a batch process I
>> need to go to each machine and manually start each job in the right
>> order,
>>
>> I was looking to resolve this problem with a tool which can manage
>> this "pipe line"
>> and I cam across several tools like Luigi and (apache-)AirFlow, I
>> started with Luigi but It didn't look
>> right for the job, and then I tried airflow, but was not able to make
>> it to work, the jobs queue never executed. =(
>>
>> Has any one have experience with airflow, or other tool like it which
>> they can recommend ?
>> My needs are to be able to execute my CURRENT shell/python/php
>> scripts and build the dependency between them, and I perfer the
>> option for remote exec so that I will have central
>> place to manage and monitor all work flow whichs are executed on
>> several nodes.
>--
>More proof the End of the World has started. Just saw this online:
>I think it's beginning! Ten minutes ago there was a group of people
>waiting at the bus stop outside my house. Now, they're all gone!
>My own blog is at https://tddpirate.zak.co.il/
>
>My opinions, as expressed in this E-mail message, are mine alone.
>They do not represent the official policy of any organization with
>which
>I may be affiliated in any way.
>WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html
>
>_______________________________________________
>Linux-il mailing list
>Linux-il@cs.huji.ac.il
>http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
_______________________________________________
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
