Re: Scaling Flink for batch jobs

2021-08-16 Thread Gorjan Todorovski
Thanks, I'll check more about job tuning. On Mon, 16 Aug 2021 at 06:28, Caizhi Weng wrote: > Hi! > > if I use parallelism of 2 or 4 - it takes the same time. >> > It might be that there is no data in some parallelisms. You can click on > the nodes in Flink web UI and see if it is the case for ea

Re: Scaling Flink for batch jobs

2021-08-15 Thread Caizhi Weng
Hi! if I use parallelism of 2 or 4 - it takes the same time. > It might be that there is no data in some parallelisms. You can click on the nodes in Flink web UI and see if it is the case for each parallelism, or you can check out the metrics of each operator. if I don't increase parallelism and

Scaling Flink for batch jobs

2021-08-13 Thread Gorjan Todorovski
Hi! I want to implement a Flink cluster as a native Kubernetes session cluster, with intention of executing Apache Beam jobs that will process only batch data, but I am not sure I understand how I would scale the cluster if I need to process large datasets. My understanding is that to be able to