Siyuan, This behavior of only periodically update metadata at the consumer is by-design to abstract users from manual metadata refresh, and having a stale metadata would not have incorrectness issues since it will only mean some newly added partitions will not be consumed for at most metadata.max.age.ms. Could you give us a bit more details in your case about why you would need immediate reflections of metadata change on your consumer change?
Guozhang On Mon, Nov 30, 2015 at 9:48 AM, Jason Gustafson <ja...@confluent.io> wrote: > Hey Siyuan, > > Are you also consuming from the topics you are querying with partitionsFor? > I'm guessing the problem is that we're pulling the metadata from a local > cache, and it's not refreshed as quick as you're expecting (the default is > 5 minutes). We only use the cache for topics which have partitions > assigned. My personal feeling is that we should probably avoid using the > cache entirely for this API. Anyway, I can think of two ways to mitigate > the problem at the moment. > > 1. (Preferred) Reduce metadata.max.age.ms in the consumer config to ensure > that metadata refreshes occur more frequently. > 2. Use a separate instance of KafkaConsumer which has no assignment to > query metadata. > > -Jason > > On Mon, Nov 30, 2015 at 8:51 AM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > Hi guys, > > > > I want to use partitionsFor method of new consumer API periodically to > get > > the monitor partition metadata change, It seems it only issue remote call > > to the server for the first time. If I add partitions after that, > > partitionsFor will return stale value. Is there a way to reuse consumer > > object to refresh the metadata change. Thanks! > > > > regards, > > Siyuan > > > -- -- Guozhang
