1. Dynamic: yeah that is sth. we could think of, this could be useful operationally. 2. Regex: I think in terms of expressiveness it should be sufficient for almost all subset of topics. In practice usually the rule of thumb is that you will create your topics that belongs to the same "group" with some prefix / suffix so that regex expression would not be crazily long.
Guozhang On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com> wrote: > something like dynamic filtering that can be updated at runtime or deny all > but allow a certain set of topics that cannot be specified easily by regex > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hmm, what kind of customized filtering do you have in mind? I thought > with > > "--whitelist" you could already specify regex to do filtering. > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > > > Hi Guozhang, > > > > > > I was meant to be topicfilter not topic-count. sorry for the confusion. > > > What I want to achieve is to pass my own customized topicfilter to MM > so > > > that I can filter out topics what ever I like. I know MM doesn't > support > > > this now. I am just thinking if this is a good feature to add in > > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > Hi Tao, > > > > > > > > Sorry I was mistaken before, yes in MM you can only directly specify > > > > "--whitelist", "--blacklist", and the number of streams you want to > > > create > > > > via "--num.streams", but cannot set specific topic-count. This is > > because > > > > MM is mainly used for cross DC replication, and hence usually will > pipe > > > all > > > > topics or a majority of the topics from the source cluster to the > > > > destination, hence usually you do not care about "some topics should > > get > > > X > > > > streams, while some other topics should get Y streams". > > > > > > > > Guozhang > > > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xiaotao...@gmail.com> > > wrote: > > > > > > > > > The topic list is not specified in consumer.properties and I don't > > > think > > > > > there is any property in consumer config that allows us to specify > > what > > > > > topics we want to consume. Can you point me to the property if > there > > is > > > > > any? > > > > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > Tao, > > > > > > > > > > > > In MM people can pass in consumer configs, in which people can > > > specify > > > > > > consumption topics, either in regular topic list format or > > whitelist > > > / > > > > > > blacklist. So I think it already does what you need? > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xiaotao...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Thank you guys for answering. I think it will be good that we > can > > > > pass > > > > > > in a > > > > > > > customised topicCount ( I think this is the interface whitelist > > and > > > > > > > backlist implement if I am not mistaken) to MM to achieve > similar > > > > thing > > > > > > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Tao, > > > > > > > > > > > > > > > > Unfortunately MM does not support whitelist / blacklist at > the > > > same > > > > > > time, > > > > > > > > and you have to choose either one upon initialization. As for > > > your > > > > > > case, > > > > > > > I > > > > > > > > think it can be captured by some reg-ex to exclude nothing > else > > > but > > > > > > "10", > > > > > > > > but I do not know the exact expression. > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < > > xiaotao...@gmail.com > > > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > > > > I actually mean if we can achieve this in mirror maker. > > > > > > > > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < > > > xiaotao...@gmail.com > > > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I have an user case where I need to consume a list topics > > > with > > > > > name > > > > > > > > that > > > > > > > > > > matches pattern topic.* except for one that is topic.10. > Is > > > > > there a > > > > > > > way > > > > > > > > > > that I can combine the use of whitelist and blacklist so > > > that I > > > > > can > > > > > > > > > achieve > > > > > > > > > > something like accept all topics with regex topic.* but > > > exclude > > > > > > > > topic.10? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Tao > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > > -- > > > Regards, > > > Tao > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Regards, > Tao > -- -- Guozhang
