Yes, that will work. message handle can filter out message sent from
certain topics

On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <j...@linkedin.com.invalid>
wrote:

> No sure if it is an option. But does filtering out topics with message
> handler works for you? Are you going to resume consuming from a topic
> after you stop consuming from it?
>
> Jiangjie (Becket) Qin
>
> On 3/12/15, 8:05 AM, "tao xiao" <xiaotao...@gmail.com> wrote:
>
> >Yes, you are right. a dynamic topicfilter is more appropriate where I can
> >filter topics at runtime via some kind of interface e.g. JMX
> >
> >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wangg...@gmail.com>
> >wrote:
> >
> >> Tao,
> >>
> >> Based on your description I think the combination of whitelist /
> >>blacklist
> >> will not achieve your goal, since it is still static.
> >>
> >> Guozhang
> >>
> >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> wrote:
> >>
> >> > Thank you Guozhang for your advice. A dynamic topic filter is what I
> >>need
> >> > so that I can stop a topic consumption when I need to at runtime.
> >> >
> >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wangg...@gmail.com>
> >> wrote:
> >> >
> >> > > 1. Dynamic: yeah that is sth. we could think of, this could be
> >>useful
> >> > > operationally.
> >> > > 2. Regex: I think in terms of expressiveness it should be sufficient
> >> for
> >> > > almost all subset of topics. In practice usually the rule of thumb
> >>is
> >> > that
> >> > > you will create your topics that belongs to the same "group" with
> >>some
> >> > > prefix / suffix so that regex expression would not be crazily long.
> >> > >
> >> > > Guozhang
> >> > >
> >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com>
> >> wrote:
> >> > >
> >> > > > something like dynamic filtering that can be updated at runtime or
> >> deny
> >> > > all
> >> > > > but allow a certain set of topics that cannot be specified easily
> >>by
> >> > > regex
> >> > > >
> >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
> >><wangg...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Hmm, what kind of customized filtering do you have in mind? I
> >> thought
> >> > > > with
> >> > > > > "--whitelist" you could already specify regex to do filtering.
> >> > > > >
> >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Guozhang,
> >> > > > > >
> >> > > > > > I was meant to be topicfilter not topic-count. sorry for the
> >> > > confusion.
> >> > > > > > What I want to achieve is to pass my own customized
> >>topicfilter
> >> to
> >> > MM
> >> > > > so
> >> > > > > > that I can filter out topics what ever I like. I know MM
> >>doesn't
> >> > > > support
> >> > > > > > this now. I am just thinking if this is a good feature to add
> >>in
> >> > > > > >
> >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
> >> wangg...@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Tao,
> >> > > > > > >
> >> > > > > > > Sorry I was mistaken before, yes in MM you can only directly
> >> > > specify
> >> > > > > > > "--whitelist", "--blacklist", and  the number of streams you
> >> want
> >> > > to
> >> > > > > > create
> >> > > > > > > via "--num.streams", but cannot set specific topic-count.
> >>This
> >> is
> >> > > > > because
> >> > > > > > > MM is mainly used for cross DC replication, and hence
> >>usually
> >> > will
> >> > > > pipe
> >> > > > > > all
> >> > > > > > > topics or a majority of the topics from the source cluster
> >>to
> >> the
> >> > > > > > > destination, hence usually you do not care about "some
> >>topics
> >> > > should
> >> > > > > get
> >> > > > > > X
> >> > > > > > > streams, while some other topics should get Y streams".
> >> > > > > > >
> >> > > > > > > Guozhang
> >> > > > > > >
> >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
> >> xiaotao...@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > The topic list is not specified in consumer.properties
> >>and I
> >> > > don't
> >> > > > > > think
> >> > > > > > > > there is any property in consumer config that allows us to
> >> > > specify
> >> > > > > what
> >> > > > > > > > topics we want to consume. Can you point me to the
> >>property
> >> if
> >> > > > there
> >> > > > > is
> >> > > > > > > > any?
> >> > > > > > > >
> >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> >> > > > wangg...@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Tao,
> >> > > > > > > > >
> >> > > > > > > > > In MM people can pass in consumer configs, in which
> >>people
> >> > can
> >> > > > > > specify
> >> > > > > > > > > consumption topics, either in regular topic list format
> >>or
> >> > > > > whitelist
> >> > > > > > /
> >> > > > > > > > > blacklist. So I think it already does what you need?
> >> > > > > > > > >
> >> > > > > > > > > Guozhang
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> >> > > xiaotao...@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Thank you guys for answering. I think it will be good
> >> that
> >> > we
> >> > > > can
> >> > > > > > > pass
> >> > > > > > > > > in a
> >> > > > > > > > > > customised topicCount ( I think this is the interface
> >> > > whitelist
> >> > > > > and
> >> > > > > > > > > > backlist implement if I am not mistaken) to MM to
> >>achieve
> >> > > > similar
> >> > > > > > > thing
> >> > > > > > > > > >
> >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> >> > > > wangg...@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Tao,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Unfortunately MM does not support whitelist /
> >>blacklist
> >> > at
> >> > > > the
> >> > > > > > same
> >> > > > > > > > > time,
> >> > > > > > > > > > > and you have to choose either one upon
> >>initialization.
> >> As
> >> > > for
> >> > > > > > your
> >> > > > > > > > > case,
> >> > > > > > > > > > I
> >> > > > > > > > > > > think it can be captured by some reg-ex to exclude
> >> > nothing
> >> > > > else
> >> > > > > > but
> >> > > > > > > > > "10",
> >> > > > > > > > > > > but I do not know the exact expression.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Guozhang
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> >> > > > > xiaotao...@gmail.com
> >> > > > > > > > > > > <javascript:;>> wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > I actually mean if we can achieve this in mirror
> >> maker.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> >> > > > > > xiaotao...@gmail.com
> >> > > > > > > > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I have an user case where I need to consume a
> >>list
> >> > > topics
> >> > > > > > with
> >> > > > > > > > name
> >> > > > > > > > > > > that
> >> > > > > > > > > > > > > matches pattern topic.* except for one that is
> >> > > topic.10.
> >> > > > Is
> >> > > > > > > > there a
> >> > > > > > > > > > way
> >> > > > > > > > > > > > > that I can combine the use of whitelist and
> >> blacklist
> >> > > so
> >> > > > > > that I
> >> > > > > > > > can
> >> > > > > > > > > > > > achieve
> >> > > > > > > > > > > > > something like accept all topics with regex
> >>topic.*
> >> > but
> >> > > > > > exclude
> >> > > > > > > > > > > topic.10?
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > --
> >> > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > Tao
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > --
> >> > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > Tao
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > -- Guozhang
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > --
> >> > > > > > > > > > Regards,
> >> > > > > > > > > > Tao
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > -- Guozhang
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Regards,
> >> > > > > > > > Tao
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > -- Guozhang
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Regards,
> >> > > > > > Tao
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > -- Guozhang
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > > Tao
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > -- Guozhang
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Tao
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
> >
> >--
> >Regards,
> >Tao
>
>


-- 
Regards,
Tao

Reply via email to