Note that with filtering in message handler, records from the source cluster are still considered as "consumed" since the offsets will be committed. If you change the filtering dynamically back to whilelist these topics, you will lose the data that gets consumed during the period of the blacklist.
Guozhang On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <[email protected]> wrote: > Yes, that will work. message handle can filter out message sent from > certain topics > > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <[email protected]> > wrote: > > > No sure if it is an option. But does filtering out topics with message > > handler works for you? Are you going to resume consuming from a topic > > after you stop consuming from it? > > > > Jiangjie (Becket) Qin > > > > On 3/12/15, 8:05 AM, "tao xiao" <[email protected]> wrote: > > > > >Yes, you are right. a dynamic topicfilter is more appropriate where I > can > > >filter topics at runtime via some kind of interface e.g. JMX > > > > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <[email protected]> > > >wrote: > > > > > >> Tao, > > >> > > >> Based on your description I think the combination of whitelist / > > >>blacklist > > >> will not achieve your goal, since it is still static. > > >> > > >> Guozhang > > >> > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <[email protected]> > wrote: > > >> > > >> > Thank you Guozhang for your advice. A dynamic topic filter is what I > > >>need > > >> > so that I can stop a topic consumption when I need to at runtime. > > >> > > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <[email protected]> > > >> wrote: > > >> > > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be > > >>useful > > >> > > operationally. > > >> > > 2. Regex: I think in terms of expressiveness it should be > sufficient > > >> for > > >> > > almost all subset of topics. In practice usually the rule of thumb > > >>is > > >> > that > > >> > > you will create your topics that belongs to the same "group" with > > >>some > > >> > > prefix / suffix so that regex expression would not be crazily > long. > > >> > > > > >> > > Guozhang > > >> > > > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <[email protected]> > > >> wrote: > > >> > > > > >> > > > something like dynamic filtering that can be updated at runtime > or > > >> deny > > >> > > all > > >> > > > but allow a certain set of topics that cannot be specified > easily > > >>by > > >> > > regex > > >> > > > > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang > > >><[email protected]> > > >> > > wrote: > > >> > > > > > >> > > > > Hmm, what kind of customized filtering do you have in mind? I > > >> thought > > >> > > > with > > >> > > > > "--whitelist" you could already specify regex to do filtering. > > >> > > > > > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao < > [email protected] > > > > > >> > > wrote: > > >> > > > > > > >> > > > > > Hi Guozhang, > > >> > > > > > > > >> > > > > > I was meant to be topicfilter not topic-count. sorry for the > > >> > > confusion. > > >> > > > > > What I want to achieve is to pass my own customized > > >>topicfilter > > >> to > > >> > MM > > >> > > > so > > >> > > > > > that I can filter out topics what ever I like. I know MM > > >>doesn't > > >> > > > support > > >> > > > > > this now. I am just thinking if this is a good feature to > add > > >>in > > >> > > > > > > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < > > >> [email protected] > > >> > > > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > > Hi Tao, > > >> > > > > > > > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only > directly > > >> > > specify > > >> > > > > > > "--whitelist", "--blacklist", and the number of streams > you > > >> want > > >> > > to > > >> > > > > > create > > >> > > > > > > via "--num.streams", but cannot set specific topic-count. > > >>This > > >> is > > >> > > > > because > > >> > > > > > > MM is mainly used for cross DC replication, and hence > > >>usually > > >> > will > > >> > > > pipe > > >> > > > > > all > > >> > > > > > > topics or a majority of the topics from the source cluster > > >>to > > >> the > > >> > > > > > > destination, hence usually you do not care about "some > > >>topics > > >> > > should > > >> > > > > get > > >> > > > > > X > > >> > > > > > > streams, while some other topics should get Y streams". > > >> > > > > > > > > >> > > > > > > Guozhang > > >> > > > > > > > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < > > >> [email protected] > > >> > > > > >> > > > > wrote: > > >> > > > > > > > > >> > > > > > > > The topic list is not specified in consumer.properties > > >>and I > > >> > > don't > > >> > > > > > think > > >> > > > > > > > there is any property in consumer config that allows us > to > > >> > > specify > > >> > > > > what > > >> > > > > > > > topics we want to consume. Can you point me to the > > >>property > > >> if > > >> > > > there > > >> > > > > is > > >> > > > > > > > any? > > >> > > > > > > > > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < > > >> > > > [email protected]> > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Tao, > > >> > > > > > > > > > > >> > > > > > > > > In MM people can pass in consumer configs, in which > > >>people > > >> > can > > >> > > > > > specify > > >> > > > > > > > > consumption topics, either in regular topic list > format > > >>or > > >> > > > > whitelist > > >> > > > > > / > > >> > > > > > > > > blacklist. So I think it already does what you need? > > >> > > > > > > > > > > >> > > > > > > > > Guozhang > > >> > > > > > > > > > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < > > >> > > [email protected] > > >> > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Thank you guys for answering. I think it will be > good > > >> that > > >> > we > > >> > > > can > > >> > > > > > > pass > > >> > > > > > > > > in a > > >> > > > > > > > > > customised topicCount ( I think this is the > interface > > >> > > whitelist > > >> > > > > and > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM to > > >>achieve > > >> > > > similar > > >> > > > > > > thing > > >> > > > > > > > > > > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < > > >> > > > [email protected]> > > >> > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Hi Tao, > > >> > > > > > > > > > > > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / > > >>blacklist > > >> > at > > >> > > > the > > >> > > > > > same > > >> > > > > > > > > time, > > >> > > > > > > > > > > and you have to choose either one upon > > >>initialization. > > >> As > > >> > > for > > >> > > > > > your > > >> > > > > > > > > case, > > >> > > > > > > > > > I > > >> > > > > > > > > > > think it can be captured by some reg-ex to exclude > > >> > nothing > > >> > > > else > > >> > > > > > but > > >> > > > > > > > > "10", > > >> > > > > > > > > > > but I do not know the exact expression. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Guozhang > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < > > >> > > > > [email protected] > > >> > > > > > > > > > > <javascript:;>> wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > > > I actually mean if we can achieve this in mirror > > >> maker. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < > > >> > > > > > [email protected] > > >> > > > > > > > > > > <javascript:;>> wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Hi, > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > I have an user case where I need to consume a > > >>list > > >> > > topics > > >> > > > > > with > > >> > > > > > > > name > > >> > > > > > > > > > > that > > >> > > > > > > > > > > > > matches pattern topic.* except for one that is > > >> > > topic.10. > > >> > > > Is > > >> > > > > > > > there a > > >> > > > > > > > > > way > > >> > > > > > > > > > > > > that I can combine the use of whitelist and > > >> blacklist > > >> > > so > > >> > > > > > that I > > >> > > > > > > > can > > >> > > > > > > > > > > > achieve > > >> > > > > > > > > > > > > something like accept all topics with regex > > >>topic.* > > >> > but > > >> > > > > > exclude > > >> > > > > > > > > > > topic.10? > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > -- > > >> > > > > > > > > > > > > Regards, > > >> > > > > > > > > > > > > Tao > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > -- > > >> > > > > > > > > > > > Regards, > > >> > > > > > > > > > > > Tao > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > -- > > >> > > > > > > > > > > -- Guozhang > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > -- > > >> > > > > > > > > > Regards, > > >> > > > > > > > > > Tao > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -- > > >> > > > > > > > > -- Guozhang > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > -- > > >> > > > > > > > Regards, > > >> > > > > > > > Tao > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > -- Guozhang > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > -- > > >> > > > > > Regards, > > >> > > > > > Tao > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > -- > > >> > > > > -- Guozhang > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > -- > > >> > > > Regards, > > >> > > > Tao > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > -- > > >> > > -- Guozhang > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Regards, > > >> > Tao > > >> > > > >> > > >> > > >> > > >> -- > > >> -- Guozhang > > >> > > > > > > > > > > > >-- > > >Regards, > > >Tao > > > > > > > -- > Regards, > Tao > -- -- Guozhang
