Re: Trouble sinking to Kafka

2022-02-23 Thread Nicolaus Weidner
Hi Marco, The documentation kind of suggestion this is the cause: > https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/connectors/kafka.html > > However, I think the documentation could benefit with a few examples and > scenarios that can ill-considered configurations. > Matthias alre

RE: Trouble sinking to Kafka

2022-02-23 Thread Schwalbe Matthias
page is very detailed … Open questions? … get back to the community 😊 Cheers Thias From: Marco Villalobos Sent: Mittwoch, 23. Februar 2022 19:11 To: Nicolaus Weidner Cc: user Subject: Re: Trouble sinking to Kafka I fixed this, but I'm not 100% sure why. Here is my theory: My check

Re: Trouble sinking to Kafka

2022-02-23 Thread Marco Villalobos
I fixed this, but I'm not 100% sure why. Here is my theory: My checkpoint interval is one minute, and the minimum pause interval is also one minute. My transaction timeout time is also one minute. I think the checkpoint causes Flink to hold the transaction open for one minute, and thus it times o

Re: Trouble sinking to Kafka

2022-02-23 Thread Nicolaus Weidner
Hi Marco, I'm no expert on the Kafka producer, but I will try to help. [1] seems to have a decent explanation of possible error causes for the error you encountered. Which leads me to two questions: if (druidProducerTransactionMaxTimeoutMs > 0) { > > properties.setProperty("transaction.max