I fixed this, but I'm not 100% sure why. Here is my theory:
My checkpoint interval is one minute, and the minimum pause interval is also one minute. My transaction timeout time is also one minute. I think the checkpoint causes Flink to hold the transaction open for one minute, and thus it times out. After I changed the *transaction.max.timeout.ms <http://transaction.max.timeout.ms>* to one hour, and the *transaction.timeout.ms <http://transaction.timeout.ms>* to five minutes, it all worked like a charm. Is my theory correct? The documentation kind of suggestion this is the cause: https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/connectors/kafka.html However, I think the documentation could benefit with a few examples and scenarios that can ill-considered configurations. Thank you. On Wed, Feb 23, 2022 at 9:29 AM Nicolaus Weidner < nicolaus.weid...@ververica.com> wrote: > Hi Marco, > > I'm no expert on the Kafka producer, but I will try to help. [1] seems to > have a decent explanation of possible error causes for the error you > encountered. > Which leads me to two questions: > > > if (druidProducerTransactionMaxTimeoutMs > 0) { >> >> properties.setProperty("transaction.max.timeout.ms", >> Integer.toString(druidProducerTransactionMaxTimeoutMs)); >> } >> if (druidProducerTransactionTimeoutMs > 0) { >> properties.setProperty("transaction.timeout.ms", >> Integer.toString(druidProducerTransactionTimeoutMs)); >> } >> >> > Have you tried increasing the timeout settings, to see if transactions > timed out? > > >> properties.setProperty("transactional.id", "local.druid"); >> >> > Do you use multiple producers (parallelism > 1)? It seems you always set > the same transactional.id, which I expect causes problems when you have > multiple producer instances (see "zombie fencing" in [2]). In that case, > just make sure they are unique. > > And one additional question: Does the error occur consistently, or only > occasionally? > > Best, > Nico > > [1] > https://stackoverflow.com/questions/53058715/what-is-reason-for-getting-producerfencedexception-during-producer-send > [2] https://stackoverflow.com/a/52304789 > >>
