That all sounds right! Usually the delay for picking up the metadata update and starting to replicate the topic won't be an issue since it's a one-time issue (around topic creation) and the time window is pretty small for that. In steady state, none of the mentioned delays would apply.
-Ewen On Thu, Jan 5, 2017 at 3:46 AM, Greenhorn Techie <[email protected]> wrote: > Thanks Ewen for your response. > > Just to summarise, here is my understanding. Apologies if something is > mis-understood. I am new to Kafka and hence still short in knowledge. > > > - MirrorMarker process automatically picks-up new topics added on the > source cluster and hence no restart of the process is needed at regular > intervals to update the list of topics > - For existing topics, MirrorMaker will replicate messages to the target > kafka cluster as and when it sees data in the source kafka cluster's > topics > - However for new topics, there might be a delay of up to (default) 5 > min i.e. metadata refresh interval to start replicating the data to the > target kafka cluster > > Please let me know if something is wrong in my understanding. > > Thanks > > On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <[email protected]> > wrote: > > > Yes, the consumer will pick up the new topics when it refreshes metadata > > (defaults to every 5 min) and start subscribing to the new topics. > > > > -Ewen > > > > On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie < > > [email protected]> > > wrote: > > > > > Hi, > > > > > > I am new to Kafka and as well as MirrorMaker. So wondering whether MM > > would > > > pick-up new topics that are created on the source cluster > automatically, > > > provided topic matches the while list pattern? > > > > > > For example, if I start MM as below, would it replicate any new topics > > that > > > are created after the MM process is launched? > > > > > > kafka-mirror-maker.sh --new.consumer --consumer.config > /opt/kafka/config/ > > > consumer.properties --producer.config /opt/kafka/config/producer. > > > properties > > > --whitelist ".*" > > > > > > Or is there a need to restart MM process when a new topic / pattern is > > > started on the source cluster? > > > > > > What should be the recommended approach in regards to this? > > > > > > Thanks > > > > > >
