Actually new MM will commit offsets even if those messages are filtered out. That¹s why I¹m asking will you resume consuming from a topic after you stop consuming from it earlier. If you are going to do this, you need do extra work in your message handler. For example, 1. When received a message that triggers stopping consumption from a topic, store the offset in message handler and start filter out the topic. 2. When received a message that triggers resuming consumption from a topic, reset the offset for that topic to the stored offset, and stop filtering.
This might need some tricks if you have multiple MM instances for synchronization. Thanks. Jiangjie (Becket) Qin On 3/12/15, 10:23 PM, "tao xiao" <xiaotao...@gmail.com> wrote: >I am not sure how MM is going to be rewritten. Based on the current >implementation in trunk offset is not committed unless it is produced to >destination. With assumption that this logic remains MM will not >acknowledge the offset back to source for filtered message. So I think it >is safe to filter messages out while keeping committed offset unchanged >for >that particular topic. Please correct me if I am wrong > >On Fri, Mar 13, 2015 at 1:12 PM, Guozhang Wang <wangg...@gmail.com> wrote: > >> Note that with filtering in message handler, records from the source >> cluster are still considered as "consumed" since the offsets will be >> committed. If you change the filtering dynamically back to whilelist >>these >> topics, you will lose the data that gets consumed during the period of >>the >> blacklist. >> >> Guozhang >> >> On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <xiaotao...@gmail.com> wrote: >> >> > Yes, that will work. message handle can filter out message sent from >> > certain topics >> > >> > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin >><j...@linkedin.com.invalid >> > >> > wrote: >> > >> > > No sure if it is an option. But does filtering out topics with >>message >> > > handler works for you? Are you going to resume consuming from a >>topic >> > > after you stop consuming from it? >> > > >> > > Jiangjie (Becket) Qin >> > > >> > > On 3/12/15, 8:05 AM, "tao xiao" <xiaotao...@gmail.com> wrote: >> > > >> > > >Yes, you are right. a dynamic topicfilter is more appropriate >>where I >> > can >> > > >filter topics at runtime via some kind of interface e.g. JMX >> > > > >> > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang >><wangg...@gmail.com> >> > > >wrote: >> > > > >> > > >> Tao, >> > > >> >> > > >> Based on your description I think the combination of whitelist / >> > > >>blacklist >> > > >> will not achieve your goal, since it is still static. >> > > >> >> > > >> Guozhang >> > > >> >> > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> >> > wrote: >> > > >> >> > > >> > Thank you Guozhang for your advice. A dynamic topic filter is >> what I >> > > >>need >> > > >> > so that I can stop a topic consumption when I need to at >>runtime. >> > > >> > >> > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang < >> wangg...@gmail.com> >> > > >> wrote: >> > > >> > >> > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could >>be >> > > >>useful >> > > >> > > operationally. >> > > >> > > 2. Regex: I think in terms of expressiveness it should be >> > sufficient >> > > >> for >> > > >> > > almost all subset of topics. In practice usually the rule of >> thumb >> > > >>is >> > > >> > that >> > > >> > > you will create your topics that belongs to the same "group" >> with >> > > >>some >> > > >> > > prefix / suffix so that regex expression would not be crazily >> > long. >> > > >> > > >> > > >> > > Guozhang >> > > >> > > >> > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao >><xiaotao...@gmail.com >> > >> > > >> wrote: >> > > >> > > >> > > >> > > > something like dynamic filtering that can be updated at >> runtime >> > or >> > > >> deny >> > > >> > > all >> > > >> > > > but allow a certain set of topics that cannot be specified >> > easily >> > > >>by >> > > >> > > regex >> > > >> > > > >> > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang >> > > >><wangg...@gmail.com> >> > > >> > > wrote: >> > > >> > > > >> > > >> > > > > Hmm, what kind of customized filtering do you have in >>mind? >> I >> > > >> thought >> > > >> > > > with >> > > >> > > > > "--whitelist" you could already specify regex to do >> filtering. >> > > >> > > > > >> > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao < >> > xiaotao...@gmail.com >> > > > >> > > >> > > wrote: >> > > >> > > > > >> > > >> > > > > > Hi Guozhang, >> > > >> > > > > > >> > > >> > > > > > I was meant to be topicfilter not topic-count. sorry >>for >> the >> > > >> > > confusion. >> > > >> > > > > > What I want to achieve is to pass my own customized >> > > >>topicfilter >> > > >> to >> > > >> > MM >> > > >> > > > so >> > > >> > > > > > that I can filter out topics what ever I like. I know >>MM >> > > >>doesn't >> > > >> > > > support >> > > >> > > > > > this now. I am just thinking if this is a good feature >>to >> > add >> > > >>in >> > > >> > > > > > >> > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < >> > > >> wangg...@gmail.com >> > > >> > > >> > > >> > > > > wrote: >> > > >> > > > > > >> > > >> > > > > > > Hi Tao, >> > > >> > > > > > > >> > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only >> > directly >> > > >> > > specify >> > > >> > > > > > > "--whitelist", "--blacklist", and the number of >>streams >> > you >> > > >> want >> > > >> > > to >> > > >> > > > > > create >> > > >> > > > > > > via "--num.streams", but cannot set specific >> topic-count. >> > > >>This >> > > >> is >> > > >> > > > > because >> > > >> > > > > > > MM is mainly used for cross DC replication, and hence >> > > >>usually >> > > >> > will >> > > >> > > > pipe >> > > >> > > > > > all >> > > >> > > > > > > topics or a majority of the topics from the source >> cluster >> > > >>to >> > > >> the >> > > >> > > > > > > destination, hence usually you do not care about >>"some >> > > >>topics >> > > >> > > should >> > > >> > > > > get >> > > >> > > > > > X >> > > >> > > > > > > streams, while some other topics should get Y >>streams". >> > > >> > > > > > > >> > > >> > > > > > > Guozhang >> > > >> > > > > > > >> > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < >> > > >> xiaotao...@gmail.com >> > > >> > > >> > > >> > > > > wrote: >> > > >> > > > > > > >> > > >> > > > > > > > The topic list is not specified in >>consumer.properties >> > > >>and I >> > > >> > > don't >> > > >> > > > > > think >> > > >> > > > > > > > there is any property in consumer config that >>allows >> us >> > to >> > > >> > > specify >> > > >> > > > > what >> > > >> > > > > > > > topics we want to consume. Can you point me to the >> > > >>property >> > > >> if >> > > >> > > > there >> > > >> > > > > is >> > > >> > > > > > > > any? >> > > >> > > > > > > > >> > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < >> > > >> > > > wangg...@gmail.com> >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > >> > > >> > > > > > > > > Tao, >> > > >> > > > > > > > > >> > > >> > > > > > > > > In MM people can pass in consumer configs, in >>which >> > > >>people >> > > >> > can >> > > >> > > > > > specify >> > > >> > > > > > > > > consumption topics, either in regular topic list >> > format >> > > >>or >> > > >> > > > > whitelist >> > > >> > > > > > / >> > > >> > > > > > > > > blacklist. So I think it already does what you >>need? >> > > >> > > > > > > > > >> > > >> > > > > > > > > Guozhang >> > > >> > > > > > > > > >> > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < >> > > >> > > xiaotao...@gmail.com >> > > >> > > > > >> > > >> > > > > > > wrote: >> > > >> > > > > > > > > >> > > >> > > > > > > > > > Thank you guys for answering. I think it will >>be >> > good >> > > >> that >> > > >> > we >> > > >> > > > can >> > > >> > > > > > > pass >> > > >> > > > > > > > > in a >> > > >> > > > > > > > > > customised topicCount ( I think this is the >> > interface >> > > >> > > whitelist >> > > >> > > > > and >> > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM >>to >> > > >>achieve >> > > >> > > > similar >> > > >> > > > > > > thing >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < >> > > >> > > > wangg...@gmail.com> >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > Hi Tao, >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / >> > > >>blacklist >> > > >> > at >> > > >> > > > the >> > > >> > > > > > same >> > > >> > > > > > > > > time, >> > > >> > > > > > > > > > > and you have to choose either one upon >> > > >>initialization. >> > > >> As >> > > >> > > for >> > > >> > > > > > your >> > > >> > > > > > > > > case, >> > > >> > > > > > > > > > I >> > > >> > > > > > > > > > > think it can be captured by some reg-ex to >> exclude >> > > >> > nothing >> > > >> > > > else >> > > >> > > > > > but >> > > >> > > > > > > > > "10", >> > > >> > > > > > > > > > > but I do not know the exact expression. >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Guozhang >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < >> > > >> > > > > xiaotao...@gmail.com >> > > >> > > > > > > > > > > <javascript:;>> wrote: >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > > I actually mean if we can achieve this in >> mirror >> > > >> maker. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao >>< >> > > >> > > > > > xiaotao...@gmail.com >> > > >> > > > > > > > > > > <javascript:;>> wrote: >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > Hi, >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > I have an user case where I need to >>consume >> a >> > > >>list >> > > >> > > topics >> > > >> > > > > > with >> > > >> > > > > > > > name >> > > >> > > > > > > > > > > that >> > > >> > > > > > > > > > > > > matches pattern topic.* except for one >>that >> is >> > > >> > > topic.10. >> > > >> > > > Is >> > > >> > > > > > > > there a >> > > >> > > > > > > > > > way >> > > >> > > > > > > > > > > > > that I can combine the use of whitelist >>and >> > > >> blacklist >> > > >> > > so >> > > >> > > > > > that I >> > > >> > > > > > > > can >> > > >> > > > > > > > > > > > achieve >> > > >> > > > > > > > > > > > > something like accept all topics with >>regex >> > > >>topic.* >> > > >> > but >> > > >> > > > > > exclude >> > > >> > > > > > > > > > > topic.10? >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > > Regards, >> > > >> > > > > > > > > > > > > Tao >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > Regards, >> > > >> > > > > > > > > > > > Tao >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > -- >> > > >> > > > > > > > > > > -- Guozhang >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > -- >> > > >> > > > > > > > > > Regards, >> > > >> > > > > > > > > > Tao >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > -- >> > > >> > > > > > > > > -- Guozhang >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > -- >> > > >> > > > > > > > Regards, >> > > >> > > > > > > > Tao >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > -- >> > > >> > > > > > > -- Guozhang >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > -- >> > > >> > > > > > Regards, >> > > >> > > > > > Tao >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > -- >> > > >> > > > > -- Guozhang >> > > >> > > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > -- >> > > >> > > > Regards, >> > > >> > > > Tao >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > -- >> > > >> > > -- Guozhang >> > > >> > > >> > > >> > >> > > >> > >> > > >> > >> > > >> > -- >> > > >> > Regards, >> > > >> > Tao >> > > >> > >> > > >> >> > > >> >> > > >> >> > > >> -- >> > > >> -- Guozhang >> > > >> >> > > > >> > > > >> > > > >> > > >-- >> > > >Regards, >> > > >Tao >> > > >> > > >> > >> > >> > -- >> > Regards, >> > Tao >> > >> >> >> >> -- >> -- Guozhang >> > > > >-- >Regards, >Tao
