Hi, I'll close KAFKA-9301 because it is duplicated by KAFKA-9312. https://issues.apache.org/jira/browse/KAFKA-9312
Tomoyuki On Mon, Dec 16, 2019 at 12:33 PM Tomoyuki Saito <aocch...@gmail.com> wrote: > > Hi, > > Filed up a JIRA. > https://issues.apache.org/jira/browse/KAFKA-9301 > > I'll look into how we can fix the issue, but I might not be able to > submit a PR very soon. > > Thanks, > Tomoyuki > > On Mon, Dec 9, 2019 at 1:52 PM Tomoyuki Saito <aocch...@gmail.com> wrote: > > > > Hi Guozhang, > > > > Thank you for your suggestion! > > I'll create a JIRA within a few days and consider submitting a PR. > > > > Thanks, > > Tomoyuki > > > > On 2019/12/09 00:37:49, Guozhang Wang <w...@gmail.com> wrote: > > > Hello Tomoyuki,> > > > > > > It seems that issue in 6494 is indeed valid, and I'd personally suggest > > > we> > > > do option 3) to fix the flush() behavior. Please feel free to create a > > > JIRA> > > > (and also submit your PR if you are interested in contributing :).> > > > > > > > > > Guozhang> > > > > > > On Sat, Dec 7, 2019 at 7:59 AM Tomoyuki Saito <ao...@gmail.com> wrote:> > > > > > > > Hi,> > > > >> > > > > ## Questions> > > > >> > > > > 1. Any possible way to make sure to avoid batch split, or oversized> > > > > batches?> > > > > 2. Any progress/discussion to fix the issue mentioned in the following> > > > > PR: https://github.com/apache/kafka/pull/6469 (kafka#6469)> > > > >> > > > > ## Background> > > > >> > > > > `FlinkKafkaProducer` expects that callbacks for sent records will be> > > > > executed after `KafkaProducer#flush`, while executing snapshot. But,> > > > > it does not hold when batch split happens, as explained in PR> > > > > kafka#6469. We are observing IllegalStateException for this issue,> > > > > with Flink deployment.> > > > >> > > > > https://github.com/apache/flink/blob/release-1.9.1/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L960> > > > >> > > > > We are looking for a workaround and a resolution for this issue.> > > > >> > > > > Possible workaround/resolution ideas:> > > > > * Avoid batch split> > > > > * Make Flink wait for inflight requests to be completed somehow> > > > > * Make `KafkaProducer#flush` block until inflight requests are> > > > > completed, even when batch split and reenqueue happens> > > > >> > > > > Thanks,> > > > > Tomoyuki> > > > >> > > > > > > > > > -- > > > > -- Guozhang> > > >
