Re: FW: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-04 Thread Kevin Lam
t;> at >> org.apache.flink.streaming.connectors.kafka.internals.FlinkKafkaInternalProducer.createProducerIdAndEpoch(FlinkKafkaInternalProducer.java:291) >> ... 20 more >> >> Any ideas? >> >> On Tue, Aug 3, 2021 at 10:34 AM Schwalbe Matthias < >> matthias

Re: FW: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-04 Thread Arvid Heise
0:34 AM Schwalbe Matthias < > matthias.schwa...@viseca.ch> wrote: > >> I don’t think that this risk is there, I guess Fabian a quick response … >> see also the later comments in the ticket. >> >> >> >> Thias >> >> >> >> *From:* Ke

Re: FW: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-04 Thread Kevin Lam
gt; > > > Thias > > > > *From:* Kevin Lam > *Sent:* Dienstag, 3. August 2021 15:56 > *To:* Schwalbe Matthias > *Cc:* user ; fabianp...@ververica.com > *Subject:* Re: FW: Error using KafkaProducer EXACTLY_ONCE semantic + > TaskManager Failure > > > > Thank you

RE: FW: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-03 Thread Schwalbe Matthias
tive ticket is solved. Hope this helps Thias From: Schwalbe Matthias Sent: Freitag, 30. Juli 2021 08:13 To: Kevin Lam mailto:kevin@shopify.com>> Subject: RE: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure Hi Kevin, I’ve had this exactly the same way. It has

FW: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-03 Thread Schwalbe Matthias
Subject: RE: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure Hi Kevin, I’ve had this exactly the same way. It has got to do with a regression bug in FlinkKafkaInternalProducer#resumeTransaction<https://github.com/apache/flink/blob/f06faf13930f2e8acccf1e04e2c250b85bdbf

Re: Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-08-03 Thread Fabian Paul
Hi Kevin, Please excuse the rather long reply duration. The behaviour you are describing is definitely worrisome. I think I need some more information and the exact stacktrace. Can you provide your checkpointing configuration i.e. frequency, how many concurrent checkpoints, rough checkpoint dur

Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure

2021-07-29 Thread Kevin Lam
Hi user@, We're developing a Flink application, and using the FlinkKafkaProducer. Semantic.EXACTLY_ONCE producer semantic to output records to a Kafka topic in an exactly-once way. We run our flink application on kubernetes. I've observed that if a task manager fails (I've simulated this by killi