Re: Intermittent Slowness with kafkaIO read

2020-08-05 Thread Mohil Khare
I also have a question that if wrong windowing is messing up the received timestamp ?? Thanks and regards Mohil On Wed, Aug 5, 2020 at 1:19 PM Mohil Khare wrote: > Just to let you know, this is how I setup kafkaIO read: > > p > > .apply("Read_From_Kafka", KafkaIO.read() > > .withBoo

Re: Intermittent Slowness with kafkaIO read

2020-08-05 Thread Mohil Khare
Just to let you know, this is how I setup kafkaIO read: p .apply("Read_From_Kafka", KafkaIO.read() .withBootstrapServers(servers) .withTopics(Arrays.asList(“topic1”, “topic2”)) .withKeyDeserializer(StringDeserializer.class) .withValueDeserializer(ByteArrayD

Intermittent Slowness with kafkaIO read

2020-08-05 Thread Mohil Khare
Hello, I am using Beam java Sdk 2.19 on dataflow. We have a system where log shipper continuously emit logs to kafka and beam read logs using KafkaIO. Sometime I am seeing slowness on kafkaIO read with one of the topics (probably during peak traffic period), where there is a 2-3 minutes between r