Re: Fink: KafkaProducer Data Loss

2017-07-31 Thread Tzu-Li (Gordon) Tai
lp. eventFilter.zip <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14522/eventFilter.zip> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14522.html Sent from

Re: Fink: KafkaProducer Data Loss

2017-07-28 Thread ninad
ata loss isn't observed frequently. 1/4 times or so. Thanks for all your help. eventFilter.zip <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14522/eventFilter.zip> -- View this message in context: http://apache-flink-user-mailing-list-archive.23

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Piotr Nowojski
>> Piotrek >> >>> On Jul 12, 2017, at 7:32 PM, ninad >> <mailto:nni...@gmail.com>> wrote: >>> >>> Hey guys, any update on this? If needed I can attach our code. >>> >>> >>> >>> -- >>> View this me

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Tzu-Li (Gordon) Tai
text: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Piotr Nowojski
t: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-07-12 Thread ninad
Hey guys, any update on this? If needed I can attach our code. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive at

Re: Fink: KafkaProducer Data Loss

2017-06-16 Thread ninad
Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13805.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-15 Thread Aljoscha Krettek
ere not replayed, and the kafka sink didn't receive all the >> messages. >> >> Attaching the files here. >> >> jmCloudera583.log >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13597/jmCloudera583.log> >> >

Re: Fink: KafkaProducer Data Loss

2017-06-11 Thread ninad
st-archive.2336050. > n4.nabble.com/file/n13597/tmOneCloudera583.log> > tmTwoCloudera583.log > <http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/file/n13597/tmTwoCloudera583.log> > > BTW, I posted my test results and logs for regular Flink v1.3.0 on

Re: Fink: KafkaProducer Data Loss

2017-06-11 Thread Tzu-Li (Gordon) Tai
Hope you guys saw that. Thanks for your patience guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13597.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-08 Thread ninad
ive an email thought. Hope you guys saw that. Thanks for your patience guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13597.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13549.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
Not yet. Planning to do that today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13544.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread Tzu-Li (Gordon) Tai
13527/tmTwo.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13527.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
wo.log <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13527/tmTwo.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13527.html Sent from the Apache Flink User Ma

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread Till Rohrmann
en email] >> <http:///user/SendEmail.jtp?type=node&node=13479&i=0>) wrote: >> >> I tested this with the standalone cluster, and I don't see this problem. >> So, >> the problem could be that we haven't built Flink against cloudera Hadoop? >

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread ninad
alone cluster, and I don't see this problem. > So, > the problem could be that we haven't built Flink against cloudera Hadoop? > I > will test it out. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nab

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread Tzu-Li (Gordon) Tai
era Hadoop? I will test it out. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13477.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread ninad
I tested this with the standalone cluster, and I don't see this problem. So, the problem could be that we haven't built Flink against cloudera Hadoop? I will test it out. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-Kaf

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
4.nabble.com/file/n13467/tmOneNoDataLoss.log> tmTwoNoDataLoss.log <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13467/tmTwoNoDataLoss.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Da

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
Thanks Till. The log files I have attached are the complete logs. They are DEBUG level. There are three files: jobManger.log, tmOne.log and tmTwo.log. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13463

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread Till Rohrmann
0b75.* > > Now, if I bring up all Kafka brokers, I am expecting that the messages > which > didn't make it to the Kafka sink should be sent. > > But that's not happening. > > All these logs are present in the files that I attached. > > I am going to try this on

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
he Kafka sink should be sent. But that's not happening. All these logs are present in the files that I attached. I am going to try this on the standalone cluster today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProd

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread Tzu-Li (Gordon) Tai
opic.event.filter=LEADER_NOT_AVAILABLE} -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13443.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-01 Thread ninad
AILABLE} -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13443.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-01 Thread Tzu-Li (Gordon) Tai
.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.snapshotState(FlinkKafkaProducerBase.java:335) > > > > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13412.html > > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-01 Thread Kostas Kloudas
Exception: Failed to send data to Kafka: Batch Expired > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:366) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.snapshotState(FlinkKafkaPro

Re: Fink: KafkaProducer Data Loss

2017-05-31 Thread ninad
:366) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.snapshotState(FlinkKafkaProducerBase.java:335) -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13412.html Sent from the Apache Flink User Mailing List archive. mail

Re: Fink: KafkaProducer Data Loss

2017-02-03 Thread ninad
Thanks, Gordon and Till. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p11431.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-02-02 Thread Tzu-Li (Gordon) Tai
ing the possible side effects of changing the above kafka settings. So, is our understanding correct? Or is there a way we can avoid this data loss by modifying some Flink settings? Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/F

Re: Fink: KafkaProducer Data Loss

2017-02-02 Thread Till Rohrmann
> We're still investigating the possible side effects of changing the above > kafka settings. > > So, is our understanding correct? Or is there a way we can avoid this data > loss by modifying some Flink settings? > > Thanks. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss- > tp11413.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Fink: KafkaProducer Data Loss

2017-02-02 Thread ninad
ka settings. So, is our understanding correct? Or is there a way we can avoid this data loss by modifying some Flink settings? Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.