Yes, that will work. message handle can filter out message sent from certain topics
On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > No sure if it is an option. But does filtering out topics with message > handler works for you? Are you going to resume consuming from a topic > after you stop consuming from it? > > Jiangjie (Becket) Qin > > On 3/12/15, 8:05 AM, "tao xiao" <xiaotao...@gmail.com> wrote: > > >Yes, you are right. a dynamic topicfilter is more appropriate where I can > >filter topics at runtime via some kind of interface e.g. JMX > > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wangg...@gmail.com> > >wrote: > > > >> Tao, > >> > >> Based on your description I think the combination of whitelist / > >>blacklist > >> will not achieve your goal, since it is still static. > >> > >> Guozhang > >> > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> wrote: > >> > >> > Thank you Guozhang for your advice. A dynamic topic filter is what I > >>need > >> > so that I can stop a topic consumption when I need to at runtime. > >> > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wangg...@gmail.com> > >> wrote: > >> > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be > >>useful > >> > > operationally. > >> > > 2. Regex: I think in terms of expressiveness it should be sufficient > >> for > >> > > almost all subset of topics. In practice usually the rule of thumb > >>is > >> > that > >> > > you will create your topics that belongs to the same "group" with > >>some > >> > > prefix / suffix so that regex expression would not be crazily long. > >> > > > >> > > Guozhang > >> > > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com> > >> wrote: > >> > > > >> > > > something like dynamic filtering that can be updated at runtime or > >> deny > >> > > all > >> > > > but allow a certain set of topics that cannot be specified easily > >>by > >> > > regex > >> > > > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang > >><wangg...@gmail.com> > >> > > wrote: > >> > > > > >> > > > > Hmm, what kind of customized filtering do you have in mind? I > >> thought > >> > > > with > >> > > > > "--whitelist" you could already specify regex to do filtering. > >> > > > > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com > > > >> > > wrote: > >> > > > > > >> > > > > > Hi Guozhang, > >> > > > > > > >> > > > > > I was meant to be topicfilter not topic-count. sorry for the > >> > > confusion. > >> > > > > > What I want to achieve is to pass my own customized > >>topicfilter > >> to > >> > MM > >> > > > so > >> > > > > > that I can filter out topics what ever I like. I know MM > >>doesn't > >> > > > support > >> > > > > > this now. I am just thinking if this is a good feature to add > >>in > >> > > > > > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < > >> wangg...@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > >> > > > > > > Hi Tao, > >> > > > > > > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only directly > >> > > specify > >> > > > > > > "--whitelist", "--blacklist", and the number of streams you > >> want > >> > > to > >> > > > > > create > >> > > > > > > via "--num.streams", but cannot set specific topic-count. > >>This > >> is > >> > > > > because > >> > > > > > > MM is mainly used for cross DC replication, and hence > >>usually > >> > will > >> > > > pipe > >> > > > > > all > >> > > > > > > topics or a majority of the topics from the source cluster > >>to > >> the > >> > > > > > > destination, hence usually you do not care about "some > >>topics > >> > > should > >> > > > > get > >> > > > > > X > >> > > > > > > streams, while some other topics should get Y streams". > >> > > > > > > > >> > > > > > > Guozhang > >> > > > > > > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < > >> xiaotao...@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > > > The topic list is not specified in consumer.properties > >>and I > >> > > don't > >> > > > > > think > >> > > > > > > > there is any property in consumer config that allows us to > >> > > specify > >> > > > > what > >> > > > > > > > topics we want to consume. Can you point me to the > >>property > >> if > >> > > > there > >> > > > > is > >> > > > > > > > any? > >> > > > > > > > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < > >> > > > wangg...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Tao, > >> > > > > > > > > > >> > > > > > > > > In MM people can pass in consumer configs, in which > >>people > >> > can > >> > > > > > specify > >> > > > > > > > > consumption topics, either in regular topic list format > >>or > >> > > > > whitelist > >> > > > > > / > >> > > > > > > > > blacklist. So I think it already does what you need? > >> > > > > > > > > > >> > > > > > > > > Guozhang > >> > > > > > > > > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < > >> > > xiaotao...@gmail.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Thank you guys for answering. I think it will be good > >> that > >> > we > >> > > > can > >> > > > > > > pass > >> > > > > > > > > in a > >> > > > > > > > > > customised topicCount ( I think this is the interface > >> > > whitelist > >> > > > > and > >> > > > > > > > > > backlist implement if I am not mistaken) to MM to > >>achieve > >> > > > similar > >> > > > > > > thing > >> > > > > > > > > > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < > >> > > > wangg...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Hi Tao, > >> > > > > > > > > > > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / > >>blacklist > >> > at > >> > > > the > >> > > > > > same > >> > > > > > > > > time, > >> > > > > > > > > > > and you have to choose either one upon > >>initialization. > >> As > >> > > for > >> > > > > > your > >> > > > > > > > > case, > >> > > > > > > > > > I > >> > > > > > > > > > > think it can be captured by some reg-ex to exclude > >> > nothing > >> > > > else > >> > > > > > but > >> > > > > > > > > "10", > >> > > > > > > > > > > but I do not know the exact expression. > >> > > > > > > > > > > > >> > > > > > > > > > > Guozhang > >> > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < > >> > > > > xiaotao...@gmail.com > >> > > > > > > > > > > <javascript:;>> wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > I actually mean if we can achieve this in mirror > >> maker. > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < > >> > > > > > xiaotao...@gmail.com > >> > > > > > > > > > > <javascript:;>> wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hi, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I have an user case where I need to consume a > >>list > >> > > topics > >> > > > > > with > >> > > > > > > > name > >> > > > > > > > > > > that > >> > > > > > > > > > > > > matches pattern topic.* except for one that is > >> > > topic.10. > >> > > > Is > >> > > > > > > > there a > >> > > > > > > > > > way > >> > > > > > > > > > > > > that I can combine the use of whitelist and > >> blacklist > >> > > so > >> > > > > > that I > >> > > > > > > > can > >> > > > > > > > > > > > achieve > >> > > > > > > > > > > > > something like accept all topics with regex > >>topic.* > >> > but > >> > > > > > exclude > >> > > > > > > > > > > topic.10? > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > -- > >> > > > > > > > > > > > > Regards, > >> > > > > > > > > > > > > Tao > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > -- > >> > > > > > > > > > > > Regards, > >> > > > > > > > > > > > Tao > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > -- > >> > > > > > > > > > > -- Guozhang > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > -- > >> > > > > > > > > > Regards, > >> > > > > > > > > > Tao > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > -- > >> > > > > > > > > -- Guozhang > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > -- > >> > > > > > > > Regards, > >> > > > > > > > Tao > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > -- Guozhang > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Regards, > >> > > > > > Tao > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > -- Guozhang > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Regards, > >> > > > Tao > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > -- Guozhang > >> > > > >> > > >> > > >> > > >> > -- > >> > Regards, > >> > Tao > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> > > > > > > > >-- > >Regards, > >Tao > > -- Regards, Tao
