For dependency management, you may want to use 'make' or modern
equivalents ('ant', 'gradle', etc.).
For controlling remote nodes, 'ansible' may be able to do the work.--- Omer Zak On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with > some room for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and > it takes 3minutes > the next job will be after start right after at ~00:05 > and so on > > the problem is that if one job fails, all other jobs which are depend > on him will fail as well, and then I get a shitload of alerts, and > the worst part is that if i have to manually start a batch process I > need to go to each machine and manually start each job in the right > order, > > I was looking to resolve this problem with a tool which can manage > this "pipe line" > and I cam across several tools like Luigi and (apache-)AirFlow, I > started with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make > it to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which > they can recommend ? > My needs are to be able to execute my CURRENT shell/python/php > scripts and build the dependency between them, and I perfer the > option for remote exec so that I will have central > place to manage and monitor all work flow whichs are executed on > several nodes. -- More proof the End of the World has started. Just saw this online: I think it's beginning! Ten minutes ago there was a group of people waiting at the bus stop outside my house. Now, they're all gone! My own blog is at https://tddpirate.zak.co.il/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html _______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
