Hi, You could try something like Concourse ( https://concourse-ci.org/ ). It allows you to define a pipeline which is comprised of jobs and the order in which they should be invoked.
-- Lior On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe <ra...@rabin.io> wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and it > takes 3minutes > the next job will be after start right after at ~00:05 > and so on > > the problem is that if one job fails, all other jobs which are depend on > him will fail as well, and then I get a shitload of alerts, and the worst > part is that if i have to manually start a batch process I need to go to > each machine and manually start each job in the right order, > > I was looking to resolve this problem with a tool which can manage this > "pipe line" > and I cam across several tools like Luigi and (apache-)AirFlow, I started > with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make it > to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which they > can recommend ? > My needs are to be able to execute my CURRENT shell/python/php scripts and > build the dependency between them, and I perfer the option for remote exec > so that I will have central > place to manage and monitor all work flow whichs are executed on several > nodes. > > Thanks in advance, > Rabin > > _______________________________________________ > Linux-il mailing list > Linux-il@cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > >
_______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
