No sure if it is an option. But does filtering out topics with message handler works for you? Are you going to resume consuming from a topic after you stop consuming from it?
Jiangjie (Becket) Qin On 3/12/15, 8:05 AM, "tao xiao" <xiaotao...@gmail.com> wrote: >Yes, you are right. a dynamic topicfilter is more appropriate where I can >filter topics at runtime via some kind of interface e.g. JMX > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wangg...@gmail.com> >wrote: > >> Tao, >> >> Based on your description I think the combination of whitelist / >>blacklist >> will not achieve your goal, since it is still static. >> >> Guozhang >> >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> wrote: >> >> > Thank you Guozhang for your advice. A dynamic topic filter is what I >>need >> > so that I can stop a topic consumption when I need to at runtime. >> > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wangg...@gmail.com> >> wrote: >> > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be >>useful >> > > operationally. >> > > 2. Regex: I think in terms of expressiveness it should be sufficient >> for >> > > almost all subset of topics. In practice usually the rule of thumb >>is >> > that >> > > you will create your topics that belongs to the same "group" with >>some >> > > prefix / suffix so that regex expression would not be crazily long. >> > > >> > > Guozhang >> > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com> >> wrote: >> > > >> > > > something like dynamic filtering that can be updated at runtime or >> deny >> > > all >> > > > but allow a certain set of topics that cannot be specified easily >>by >> > > regex >> > > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang >><wangg...@gmail.com> >> > > wrote: >> > > > >> > > > > Hmm, what kind of customized filtering do you have in mind? I >> thought >> > > > with >> > > > > "--whitelist" you could already specify regex to do filtering. >> > > > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com> >> > > wrote: >> > > > > >> > > > > > Hi Guozhang, >> > > > > > >> > > > > > I was meant to be topicfilter not topic-count. sorry for the >> > > confusion. >> > > > > > What I want to achieve is to pass my own customized >>topicfilter >> to >> > MM >> > > > so >> > > > > > that I can filter out topics what ever I like. I know MM >>doesn't >> > > > support >> > > > > > this now. I am just thinking if this is a good feature to add >>in >> > > > > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < >> wangg...@gmail.com >> > > >> > > > > wrote: >> > > > > > >> > > > > > > Hi Tao, >> > > > > > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only directly >> > > specify >> > > > > > > "--whitelist", "--blacklist", and the number of streams you >> want >> > > to >> > > > > > create >> > > > > > > via "--num.streams", but cannot set specific topic-count. >>This >> is >> > > > > because >> > > > > > > MM is mainly used for cross DC replication, and hence >>usually >> > will >> > > > pipe >> > > > > > all >> > > > > > > topics or a majority of the topics from the source cluster >>to >> the >> > > > > > > destination, hence usually you do not care about "some >>topics >> > > should >> > > > > get >> > > > > > X >> > > > > > > streams, while some other topics should get Y streams". >> > > > > > > >> > > > > > > Guozhang >> > > > > > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < >> xiaotao...@gmail.com >> > > >> > > > > wrote: >> > > > > > > >> > > > > > > > The topic list is not specified in consumer.properties >>and I >> > > don't >> > > > > > think >> > > > > > > > there is any property in consumer config that allows us to >> > > specify >> > > > > what >> > > > > > > > topics we want to consume. Can you point me to the >>property >> if >> > > > there >> > > > > is >> > > > > > > > any? >> > > > > > > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < >> > > > wangg...@gmail.com> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Tao, >> > > > > > > > > >> > > > > > > > > In MM people can pass in consumer configs, in which >>people >> > can >> > > > > > specify >> > > > > > > > > consumption topics, either in regular topic list format >>or >> > > > > whitelist >> > > > > > / >> > > > > > > > > blacklist. So I think it already does what you need? >> > > > > > > > > >> > > > > > > > > Guozhang >> > > > > > > > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < >> > > xiaotao...@gmail.com >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Thank you guys for answering. I think it will be good >> that >> > we >> > > > can >> > > > > > > pass >> > > > > > > > > in a >> > > > > > > > > > customised topicCount ( I think this is the interface >> > > whitelist >> > > > > and >> > > > > > > > > > backlist implement if I am not mistaken) to MM to >>achieve >> > > > similar >> > > > > > > thing >> > > > > > > > > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < >> > > > wangg...@gmail.com> >> > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Hi Tao, >> > > > > > > > > > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / >>blacklist >> > at >> > > > the >> > > > > > same >> > > > > > > > > time, >> > > > > > > > > > > and you have to choose either one upon >>initialization. >> As >> > > for >> > > > > > your >> > > > > > > > > case, >> > > > > > > > > > I >> > > > > > > > > > > think it can be captured by some reg-ex to exclude >> > nothing >> > > > else >> > > > > > but >> > > > > > > > > "10", >> > > > > > > > > > > but I do not know the exact expression. >> > > > > > > > > > > >> > > > > > > > > > > Guozhang >> > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < >> > > > > xiaotao...@gmail.com >> > > > > > > > > > > <javascript:;>> wrote: >> > > > > > > > > > > >> > > > > > > > > > > > I actually mean if we can achieve this in mirror >> maker. >> > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < >> > > > > > xiaotao...@gmail.com >> > > > > > > > > > > <javascript:;>> wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi, >> > > > > > > > > > > > > >> > > > > > > > > > > > > I have an user case where I need to consume a >>list >> > > topics >> > > > > > with >> > > > > > > > name >> > > > > > > > > > > that >> > > > > > > > > > > > > matches pattern topic.* except for one that is >> > > topic.10. >> > > > Is >> > > > > > > > there a >> > > > > > > > > > way >> > > > > > > > > > > > > that I can combine the use of whitelist and >> blacklist >> > > so >> > > > > > that I >> > > > > > > > can >> > > > > > > > > > > > achieve >> > > > > > > > > > > > > something like accept all topics with regex >>topic.* >> > but >> > > > > > exclude >> > > > > > > > > > > topic.10? >> > > > > > > > > > > > > >> > > > > > > > > > > > > -- >> > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > Tao >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > -- >> > > > > > > > > > > > Regards, >> > > > > > > > > > > > Tao >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > -- >> > > > > > > > > > > -- Guozhang >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Regards, >> > > > > > > > > > Tao >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > -- >> > > > > > > > > -- Guozhang >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Regards, >> > > > > > > > Tao >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > -- Guozhang >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Regards, >> > > > > > Tao >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > -- Guozhang >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Regards, >> > > > Tao >> > > > >> > > >> > > >> > > >> > > -- >> > > -- Guozhang >> > > >> > >> > >> > >> > -- >> > Regards, >> > Tao >> > >> >> >> >> -- >> -- Guozhang >> > > > >-- >Regards, >Tao
