Hello Boyang, Just realized one thing about timeout configurations that we should consider including in this KIP as well:
1) In Producer we have: max.block.ms (default value 60sec), request.timeout (30sec), delivery.timeout.ms (120sec), transaction.timeout (60sec) 2) In Consumer we have: session.timeout (10sec), request.timeout (30sec), default.api.timeout.ms (60sec). Within a transaction (i.e. after we've beginTxn), we could potentially call consumer blocking APIs that depend on default.api.timeout.ms, and call producer blocking APIs that depend on max.block.ms. Also, if the user is following a consumer->producer pattern, then it could be kicked and fenced either by txn or by consumer group session. So we want to make sure that in the caller, e.g. Kafka Streams: 1) transaction.timeout < max.block.ms 2) transaction.timeout < delivery.timeout.ms 3) transaction.timeout < default.api.timeout.ms 4) transaction.timeout ~= default.api.timeout.ms (I think this is already mentioned in the KIP, just wanted to bring this up again) We do not need to override the default since not every users are following the consumer -> producer pattern, but in cases like Streams where it is indeed the case, we should override the default values to obey the above rules. Guozhang On Thu, Sep 5, 2019 at 5:47 PM Guozhang Wang <wangg...@gmail.com> wrote: > Thanks Boyang, I'm +1 on the KIP. > > Could you also update the detailed design doc > https://docs.google.com/document/d/1LhzHGeX7_Lay4xvrEXxfciuDWATjpUXQhrEIkph9qRE/edit > which > seems a bit out-dated with the latest proposal? > > > Guozhang > > On Wed, Sep 4, 2019 at 10:45 AM Boyang Chen <reluctanthero...@gmail.com> > wrote: > >> Hey all, >> >> I would like to start the vote for KIP-447 >> < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics >> >. >> This is a very important step to improve Kafka Streams scalability in >> exactly-once semantics, by avoiding linearly increasing number of >> producers >> with topic partition increases. >> >> Thanks, >> Boyang >> > > > -- > -- Guozhang > -- -- Guozhang
