What have you tried so far to increase performance? (Did you try
different combinations of -yn and -ys?)
Can you provide us with your application? What source/sink are you using?
On 08.08.2018 07:59, Ravi Bhushan Ratnakar wrote:
Hi Everybody,
Currently I am working on a project where i need to write a Flink
Batch Application which has to process hourly data around 400GB of
compressed sequence file. After processing, it has write it as
compressed parquet format in S3.
I have managed to write the application in Flink and able to run
successfully process the whole hour data and write in Parquet format
in S3. But the problem is this that it is not able to meet the
performance of the existing application which is written using Spark
Batch(running in production).
Current Spark Batch
Cluster size - Aws EMR - 1 Master + 100 worker node of m4.4xlarge (
16vCpu, 64GB RAM), each instance with 160GB disk volume
Input data - Around 400GB
Time Taken to process - Around 36 mins
------------------------------------------------------------
Flink Batch
Cluster size - Aws EMR - 1 Master + 100 worker node of r4.4xlarge (
16vCpu, 64GB RAM), each instance with 630GB disk volume
Transient Job - flink run -m yarn-cluster -yn 792 -ys 2 -ytm 14000
-yjm 114736
Input data - Around 400GB
Time Taken to process - Around 1 hour
I have given all the node memory to jobmanager just to make sure that
there is a dedicated node for jobmanager so that it doesn't face any
issue related to resources.
We are already running Flink Batch job with double RAM compare to
Spark Batch however we are not able get the same performance.
Kindly suggest on this to achieve the same performance as we are
getting from Spark Batch
Thanks,
Ravi
