Apache Airflow is a scheduling system that can help manage data pipelines. I have seen Airflow is used to manage a few thousand hive/spark/presto pipelines.
-Rui On Fri, Feb 8, 2019 at 4:08 PM Sridevi Nookala < snook...@parallelwireless.com> wrote: > Hi, > > > Our analytics app has many data pipelines , some in python /java (using > beam) etc, > > Any suggestions for a pipeline manager/scheduler framework that > manages/orchestrates these different pipelines. > > > thanks > > Sri >
