Hello Brian, Thanks for the KIP.
I think using asynchronous metadata update to address 1) metadata update blocking send, but for other issues, currently at producer we do have a configurable `METADATA_MAX_AGE_CONFIG` similar to consumer, by default is 5min. So maybe we do not need to introduce new configs here, but only change the semantics of that config from global expiry (today we just enforce a full metadata update for the whole cluster) to single-topic expiry, and we can also extend its expiry deadline whenever that metadata is successfully used to send a produce request. Guozhang On Thu, Oct 3, 2019 at 6:51 PM Lucas Bradstreet <lu...@confluent.io> wrote: > Hi Brian, > > This looks great, and should help reduce blocking and high metadata request > volumes when the producer is sending to large numbers of topics, especially > at low volumes. I think the approach to make metadata fetching asynchronous > and batch metadata requests together will help significantly. > > The only other approach I can think of is to allow users to supply the > producer with the expected topics upfront, allowing the producer to perform > a single initial metadata request before any sends occur. I see no real > advantages to this approach compared to the async method you’ve proposed, > but maybe we could add it to the rejected alternatives section? > > Thanks, > > Lucas > > On Fri, 20 Sep 2019 at 11:46, Brian Byrne <bby...@confluent.io> wrote: > > > I've updated the 'Proposed Changes' to include two new producer > > configuration variables: topic.expiry.ms and topic.refresh.ms. Please > take > > a look. > > > > Thanks, > > Brian > > > > On Tue, Sep 17, 2019 at 12:59 PM Brian Byrne <bby...@confluent.io> > wrote: > > > > > Dev team, > > > > > > Requesting discussion for improvement to the producer when dealing > with a > > > large number of topics. > > > > > > KIP: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8904 > > > > > > Thoughts and feedback would be appreciated. > > > > > > Thanks, > > > Brian > > > > > > -- -- Guozhang
