Re: Fink: KafkaProducer Data Loss

Hi Gordon, I was able to reproduce the data loss on standalone flink cluster also. I have stripped down version of our code with here: Environment: Flink standalone 1.3.0 Kafka 0.9 *What the code is doing:* -consume messages from kafka topic ('event.filter.topic' property in application.propertie

Re: FlinkKafkaConsumer subscribes to partitions in restoredState only.

Got it. Thanks Gordon. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumer-subscribes-to-partitions-in-restoredState-only-tp14233p14484.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble

Re: FlinkKafkaConsumer subscribes to partitions in restoredState only.

Any update on this guys? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumer-subscribes-to-partitions-in-restoredState-only-tp14233p14410.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabbl

FlinkKafkaConsumer subscribes to partitions in restoredState only.

Hello, We're noticing that FlinkKafkaConsumer subscribes to partitions in restored state only. Thus, partitions which aren't in restored state aren't read. We have to restart the job, for FlinkKafkaConsumer to read from all partitions. Here are the details: Environment: Flink-1.3.0, standalone c

Re: Fink: KafkaProducer Data Loss

Hey guys, any update on this? If needed I can attach our code. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabb

Re: Fink: KafkaProducer Data Loss

Hi Aljoscha, I gather you guys aren't able to reproduce this. Here are the answers to your questions: How do you ensure that you only shut down the brokers once Flink has read all the data that you expect it to read Ninad: I am able to see the number of messages received on the Flink J

Re: Fink: KafkaProducer Data Loss

Thanks Gordon. On Jun 11, 2017 9:11 AM, "Tzu-Li (Gordon) Tai [via Apache Flink User Mailing List archive.]" wrote: > Hi Ninad, > > Thanks for the logs! > Just to let you know, I’ll continue to investigate this early next week. > > Cheers, > Gordon > >

Re: Fink: KafkaProducer Data Loss

I built Flink v1.3.0 with cloudera hadoop and am able to see the data loss. Here are the details: *tmOneCloudera583.log* Received session window task: *2017-06-08 15:10:46,131 INFO org.apache.flink.runtime.taskmanager.Task - TriggerWindow(ProcessingTimeSessionWindows(3

Re: Fink: KafkaProducer Data Loss

I tried to build Flink with hadoop cdc 5.8.3 and test it, but it was unable to come up. When I ran the yarn-session script, it exited with something like "Can't get the url for job manager". I didn't spend much time figuring out what's wrong, and went straight to Flink 1.3.0. I ran several tests,

Re: Fink: KafkaProducer Data Loss

Not yet. Planning to do that today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13544.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

Hi Till, Attaching the logs to this post again. Thanks. jobManager.log tmOne.log tmTwo.log

Re: Fink: KafkaProducer Data Loss

; another look at the Kafka sink code, I’m pretty sure this shouldn’t be the > case. > > Many thanks, > Gordon > > On 4 June 2017 at 2:14:40 PM, ninad ([hidden email] > <http:///user/SendEmail.jtp?type=node&node=13479&i=0>) wrote: > > I tested this with the stand

Re: Fink: KafkaProducer Data Loss

I tested this with the standalone cluster, and I don't see this problem. So, the problem could be that we haven't built Flink against cloudera Hadoop? I will test it out. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data

Re: Fink: KafkaProducer Data Loss

I ran a few tests and was able to find the case where there won't be a data loss. And here's how the two tests are different. *The case where data loss is observed:* 1) Kafka source receives data. (Session window trigger hasn't been fired yet.) 2) Bring all Kafka brokers down. 3) Flink trigge

Re: Fink: KafkaProducer Data Loss

Thanks Till. The log files I have attached are the complete logs. They are DEBUG level. There are three files: jobManger.log, tmOne.log and tmTwo.log. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13463.

Re: Fink: KafkaProducer Data Loss

Thanks Gordon. *2017-06-01 20:22:44,400 WARN org.apache.kafka.clients.producer.internals.Sender - Got error produce response with correlation id 4 on topic-partit ion topic.http.stream.event.processor-0, retrying (9 attempts left). Error: NOT_ENOUGH_REPLICAS , not sure if this may be rel

Re: Fink: KafkaProducer Data Loss

Thanks Gordon and Kostas. Gordon, "When a failure occurs in the job, Flink uses the last completed checkpoint to restart the job. In the case of the Flink Kafka producer, this essentially makes sure that records which did not make it into Kafka and caused the last run to fail are reprocessed and

Re: Fink: KafkaProducer Data Loss

Thanks for the fix guys. I am trying to test this with 1.1.5, but still seeing a data loss. I am not able to get much from logs except this: Here's our use case: 1) Consume from Kafka 2) Apply session window 3) Send messages of window to Kafka If there's a failure in step 3, because all kafka br

Re: Fink: KafkaProducer Data Loss

Thanks, Gordon and Till. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p11431.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Fink: KafkaProducer Data Loss

Our Flink streaming workflow publishes messages to Kafka. KafkaProducer's 'retry' mechanism doesn't kick in until a message is added to it's internal buffer. If there's an exception before that, KafkaProducer will throw that exception, and seems like Flink isn't handling that. In this case there w