Hi Amit,
how is the memory consumption when the jobs get stuck? Is the Java GC
active? Are you using off-heap memory?
Regards,
Timo
Am 2/12/18 um 10:10 AM schrieb Amit Jain:
Hi,
We have created Batch job where we are trying to merge set of S3
directories in TextFormat with the old snapshot in Parquet format.
We are running 50 such jobs daily and found the progress of few random jobs
get stuck in between. We have gone through logs of JobManager, TaskManager
and could not get any useful information there.
Important operators involved, are read using TextInputFormat, read using
HadoopInputFormat, FullOuterJoin, write using our BucketingSink code.
Please help resolve this issue.
Flink Version 1.3.2 deployed on Yarn Container
--
Thanks,
Amit
