Re: "Slow ReadProcessor" warnings when using BucketSink

2016-10-26 Thread Robert Metzger
Hi Max, maybe you need to ask this question on the Hadoop user mailing list (or your Hadoop vendor support, if you are using a Hadoop distribution). On Tue, Oct 18, 2016 at 11:19 AM, static-max wrote: > Hi Robert, > > thanks for your reply. I also didn't find anything helpful on Google. > > I c

Re: "Slow ReadProcessor" warnings when using BucketSink

2016-10-18 Thread static-max
Hi Robert, thanks for your reply. I also didn't find anything helpful on Google. I checked all GC Times, they look OK. Here are GC Times for the Job Manager (the job is running fine since 5 days): Collector Count Time PS-MarkSweep 3 1s PS-Scavenge 5814 2m 12s I have no window or any computation

Re: "Slow ReadProcessor" warnings when using BucketSink

2016-10-12 Thread Robert Metzger
Hi, I haven't seen this error before. Also, I didn't find anything helpful searching for the error on Google. Did you check the GC times also for Flink? Is your Flink job doing any heavy tasks (like maintaining large windows, or other operations involving a lot of heap space?) Regards, Robert O

"Slow ReadProcessor" warnings when using BucketSink

2016-10-11 Thread static-max
Hi, I have a low throughput job (approx. 1000 messager per Minute), that consumes from Kafka und writes directly to HDFS. After an hour or so, I get the following warnings in the Task Manager log: 2016-10-10 01:59:44,635 WARN org.apache.hadoop.hdfs.DFSClient - Slow ReadProcessor