Lucas, Guozhang, Thank you for the comments. Good point on METADATA_MAX_AGE_CONFIG - it looks like the ProducerMetadata was differentiating between expiry and refresh, but it should be unnecessary to do so once the cost of fetching a single topic's metadata is reduced.
I've updated the rejected alternatives and removed the config variables. Brian On Fri, Oct 4, 2019 at 9:20 AM Guozhang Wang <wangg...@gmail.com> wrote: > Hello Brian, > > Thanks for the KIP. > > I think using asynchronous metadata update to address 1) metadata update > blocking send, but for other issues, currently at producer we do have a > configurable `METADATA_MAX_AGE_CONFIG` similar to consumer, by default is > 5min. So maybe we do not need to introduce new configs here, but only > change the semantics of that config from global expiry (today we just > enforce a full metadata update for the whole cluster) to single-topic > expiry, and we can also extend its expiry deadline whenever that metadata > is successfully used to send a produce request. > > > Guozhang > > > > On Thu, Oct 3, 2019 at 6:51 PM Lucas Bradstreet <lu...@confluent.io> > wrote: > > > Hi Brian, > > > > This looks great, and should help reduce blocking and high metadata > request > > volumes when the producer is sending to large numbers of topics, > especially > > at low volumes. I think the approach to make metadata fetching > asynchronous > > and batch metadata requests together will help significantly. > > > > The only other approach I can think of is to allow users to supply the > > producer with the expected topics upfront, allowing the producer to > perform > > a single initial metadata request before any sends occur. I see no real > > advantages to this approach compared to the async method you’ve proposed, > > but maybe we could add it to the rejected alternatives section? > > > > Thanks, > > > > Lucas > > > > On Fri, 20 Sep 2019 at 11:46, Brian Byrne <bby...@confluent.io> wrote: > > > > > I've updated the 'Proposed Changes' to include two new producer > > > configuration variables: topic.expiry.ms and topic.refresh.ms. Please > > take > > > a look. > > > > > > Thanks, > > > Brian > > > > > > On Tue, Sep 17, 2019 at 12:59 PM Brian Byrne <bby...@confluent.io> > > wrote: > > > > > > > Dev team, > > > > > > > > Requesting discussion for improvement to the producer when dealing > > with a > > > > large number of topics. > > > > > > > > KIP: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8904 > > > > > > > > Thoughts and feedback would be appreciated. > > > > > > > > Thanks, > > > > Brian > > > > > > > > > > > > -- > -- Guozhang >
