Actually new MM will commit offsets even if those messages are filtered out. That¹s why I¹m asking will you resume consuming from a topic after you stop consuming from it earlier. If you are going to do this, you need do extra work in your message handler. For example, 1. When received a message that triggers stopping consumption from a topic, store the offset in message handler and start filter out the topic. 2. When received a message that triggers resuming consumption from a topic, reset the offset for that topic to the stored offset, and stop filtering.
This might need some tricks if you have multiple MM instances for synchronization. Thanks. Jiangjie (Becket) Qin On 3/12/15, 10:23 PM, "tao xiao" <[email protected]> wrote: >I am not sure how MM is going to be rewritten. Based on the current >implementation in trunk offset is not committed unless it is produced to >destination. With assumption that this logic remains MM will not >acknowledge the offset back to source for filtered message. So I think it >is safe to filter messages out while keeping committed offset unchanged >for >that particular topic. Please correct me if I am wrong > >On Fri, Mar 13, 2015 at 1:12 PM, Guozhang Wang <[email protected]> wrote: > >> Note that with filtering in message handler, records from the source >> cluster are still considered as "consumed" since the offsets will be >> committed. If you change the filtering dynamically back to whilelist >>these >> topics, you will lose the data that gets consumed during the period of >>the >> blacklist. >> >> Guozhang >> >> On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <[email protected]> wrote: >> >> > Yes, that will work. message handle can filter out message sent from >> > certain topics >> > >> > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin >><[email protected] >> > >> > wrote: >> > >> > > No sure if it is an option. But does filtering out topics with >>message >> > > handler works for you? Are you going to resume consuming from a >>topic >> > > after you stop consuming from it? >> > > >> > > Jiangjie (Becket) Qin >> > > >> > > On 3/12/15, 8:05 AM, "tao xiao" <[email protected]> wrote: >> > > >> > > >Yes, you are right. a dynamic topicfilter is more appropriate >>where I >> > can >> > > >filter topics at runtime via some kind of interface e.g. JMX >> > > > >> > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang >><[email protected]> >> > > >wrote: >> > > > >> > > >> Tao, >> > > >> >> > > >> Based on your description I think the combination of whitelist / >> > > >>blacklist >> > > >> will not achieve your goal, since it is still static. >> > > >> >> > > >> Guozhang >> > > >> >> > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <[email protected]> >> > wrote: >> > > >> >> > > >> > Thank you Guozhang for your advice. A dynamic topic filter is >> what I >> > > >>need >> > > >> > so that I can stop a topic consumption when I need to at >>runtime. >> > > >> > >> > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang < >> [email protected]> >> > > >> wrote: >> > > >> > >> > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could >>be >> > > >>useful >> > > >> > > operationally. >> > > >> > > 2. Regex: I think in terms of expressiveness it should be >> > sufficient >> > > >> for >> > > >> > > almost all subset of topics. In practice usually the rule of >> thumb >> > > >>is >> > > >> > that >> > > >> > > you will create your topics that belongs to the same "group" >> with >> > > >>some >> > > >> > > prefix / suffix so that regex expression would not be crazily >> > long. >> > > >> > > >> > > >> > > Guozhang >> > > >> > > >> > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao >><[email protected] >> > >> > > >> wrote: >> > > >> > > >> > > >> > > > something like dynamic filtering that can be updated at >> runtime >> > or >> > > >> deny >> > > >> > > all >> > > >> > > > but allow a certain set of topics that cannot be specified >> > easily >> > > >>by >> > > >> > > regex >> > > >> > > > >> > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang >> > > >><[email protected]> >> > > >> > > wrote: >> > > >> > > > >> > > >> > > > > Hmm, what kind of customized filtering do you have in >>mind? >> I >> > > >> thought >> > > >> > > > with >> > > >> > > > > "--whitelist" you could already specify regex to do >> filtering. >> > > >> > > > > >> > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao < >> > [email protected] >> > > > >> > > >> > > wrote: >> > > >> > > > > >> > > >> > > > > > Hi Guozhang, >> > > >> > > > > > >> > > >> > > > > > I was meant to be topicfilter not topic-count. sorry >>for >> the >> > > >> > > confusion. >> > > >> > > > > > What I want to achieve is to pass my own customized >> > > >>topicfilter >> > > >> to >> > > >> > MM >> > > >> > > > so >> > > >> > > > > > that I can filter out topics what ever I like. I know >>MM >> > > >>doesn't >> > > >> > > > support >> > > >> > > > > > this now. I am just thinking if this is a good feature >>to >> > add >> > > >>in >> > > >> > > > > > >> > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < >> > > >> [email protected] >> > > >> > > >> > > >> > > > > wrote: >> > > >> > > > > > >> > > >> > > > > > > Hi Tao, >> > > >> > > > > > > >> > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only >> > directly >> > > >> > > specify >> > > >> > > > > > > "--whitelist", "--blacklist", and the number of >>streams >> > you >> > > >> want >> > > >> > > to >> > > >> > > > > > create >> > > >> > > > > > > via "--num.streams", but cannot set specific >> topic-count. >> > > >>This >> > > >> is >> > > >> > > > > because >> > > >> > > > > > > MM is mainly used for cross DC replication, and hence >> > > >>usually >> > > >> > will >> > > >> > > > pipe >> > > >> > > > > > all >> > > >> > > > > > > topics or a majority of the topics from the source >> cluster >> > > >>to >> > > >> the >> > > >> > > > > > > destination, hence usually you do not care about >>"some >> > > >>topics >> > > >> > > should >> > > >> > > > > get >> > > >> > > > > > X >> > > >> > > > > > > streams, while some other topics should get Y >>streams". >> > > >> > > > > > > >> > > >> > > > > > > Guozhang >> > > >> > > > > > > >> > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < >> > > >> [email protected] >> > > >> > > >> > > >> > > > > wrote: >> > > >> > > > > > > >> > > >> > > > > > > > The topic list is not specified in >>consumer.properties >> > > >>and I >> > > >> > > don't >> > > >> > > > > > think >> > > >> > > > > > > > there is any property in consumer config that >>allows >> us >> > to >> > > >> > > specify >> > > >> > > > > what >> > > >> > > > > > > > topics we want to consume. Can you point me to the >> > > >>property >> > > >> if >> > > >> > > > there >> > > >> > > > > is >> > > >> > > > > > > > any? >> > > >> > > > > > > > >> > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < >> > > >> > > > [email protected]> >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > >> > > >> > > > > > > > > Tao, >> > > >> > > > > > > > > >> > > >> > > > > > > > > In MM people can pass in consumer configs, in >>which >> > > >>people >> > > >> > can >> > > >> > > > > > specify >> > > >> > > > > > > > > consumption topics, either in regular topic list >> > format >> > > >>or >> > > >> > > > > whitelist >> > > >> > > > > > / >> > > >> > > > > > > > > blacklist. So I think it already does what you >>need? >> > > >> > > > > > > > > >> > > >> > > > > > > > > Guozhang >> > > >> > > > > > > > > >> > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < >> > > >> > > [email protected] >> > > >> > > > > >> > > >> > > > > > > wrote: >> > > >> > > > > > > > > >> > > >> > > > > > > > > > Thank you guys for answering. I think it will >>be >> > good >> > > >> that >> > > >> > we >> > > >> > > > can >> > > >> > > > > > > pass >> > > >> > > > > > > > > in a >> > > >> > > > > > > > > > customised topicCount ( I think this is the >> > interface >> > > >> > > whitelist >> > > >> > > > > and >> > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM >>to >> > > >>achieve >> > > >> > > > similar >> > > >> > > > > > > thing >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < >> > > >> > > > [email protected]> >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > Hi Tao, >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / >> > > >>blacklist >> > > >> > at >> > > >> > > > the >> > > >> > > > > > same >> > > >> > > > > > > > > time, >> > > >> > > > > > > > > > > and you have to choose either one upon >> > > >>initialization. >> > > >> As >> > > >> > > for >> > > >> > > > > > your >> > > >> > > > > > > > > case, >> > > >> > > > > > > > > > I >> > > >> > > > > > > > > > > think it can be captured by some reg-ex to >> exclude >> > > >> > nothing >> > > >> > > > else >> > > >> > > > > > but >> > > >> > > > > > > > > "10", >> > > >> > > > > > > > > > > but I do not know the exact expression. >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Guozhang >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < >> > > >> > > > > [email protected] >> > > >> > > > > > > > > > > <javascript:;>> wrote: >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > > I actually mean if we can achieve this in >> mirror >> > > >> maker. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao >>< >> > > >> > > > > > [email protected] >> > > >> > > > > > > > > > > <javascript:;>> wrote: >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > Hi, >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > I have an user case where I need to >>consume >> a >> > > >>list >> > > >> > > topics >> > > >> > > > > > with >> > > >> > > > > > > > name >> > > >> > > > > > > > > > > that >> > > >> > > > > > > > > > > > > matches pattern topic.* except for one >>that >> is >> > > >> > > topic.10. >> > > >> > > > Is >> > > >> > > > > > > > there a >> > > >> > > > > > > > > > way >> > > >> > > > > > > > > > > > > that I can combine the use of whitelist >>and >> > > >> blacklist >> > > >> > > so >> > > >> > > > > > that I >> > > >> > > > > > > > can >> > > >> > > > > > > > > > > > achieve >> > > >> > > > > > > > > > > > > something like accept all topics with >>regex >> > > >>topic.* >> > > >> > but >> > > >> > > > > > exclude >> > > >> > > > > > > > > > > topic.10? >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > > Regards, >> > > >> > > > > > > > > > > > > Tao >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > Regards, >> > > >> > > > > > > > > > > > Tao >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > -- >> > > >> > > > > > > > > > > -- Guozhang >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > -- >> > > >> > > > > > > > > > Regards, >> > > >> > > > > > > > > > Tao >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > -- >> > > >> > > > > > > > > -- Guozhang >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > -- >> > > >> > > > > > > > Regards, >> > > >> > > > > > > > Tao >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > -- >> > > >> > > > > > > -- Guozhang >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > -- >> > > >> > > > > > Regards, >> > > >> > > > > > Tao >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > -- >> > > >> > > > > -- Guozhang >> > > >> > > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > -- >> > > >> > > > Regards, >> > > >> > > > Tao >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > -- >> > > >> > > -- Guozhang >> > > >> > > >> > > >> > >> > > >> > >> > > >> > >> > > >> > -- >> > > >> > Regards, >> > > >> > Tao >> > > >> > >> > > >> >> > > >> >> > > >> >> > > >> -- >> > > >> -- Guozhang >> > > >> >> > > > >> > > > >> > > > >> > > >-- >> > > >Regards, >> > > >Tao >> > > >> > > >> > >> > >> > -- >> > Regards, >> > Tao >> > >> >> >> >> -- >> -- Guozhang >> > > > >-- >Regards, >Tao
