Acked. `ordering_key` looks good to me. - Sijie
On Mon, Apr 8, 2019 at 11:40 AM Jia Zhai <zhaiji...@gmail.com> wrote: > Thanks Penghui & Sijie for the comments, updated the wiki link. > > 1. we should add the new ordering_key, it is as Penghui mentioned, There > may features that all enabled, but want to use key for different purposes. > 2. regarding consumer listener in "failover", it is originally for > failover/exclusive mode, which only let one consumer alive. In > Key_Failover, it is more like shared mode, all alive consumer should be > able to consume messages. > > Best Regards. > > > Jia Zhai > > Beijing, China > > Mobile: +86 15810491983 > > > > > On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <guosi...@gmail.com> wrote: > > > Penguin, > > > > Thank you for your clarification! That makes sense to me then. > > > > Sijie > > > > On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <codelipeng...@gmail.com> > wrote: > > > > > @Sijie Guo <si...@apache.org> > > > > > > Sorry, I may not have described my point in the previous email. > > > I try to describe it again. :) > > > > > > User can use key to achieve business logic(e.g. use user_id as message > > key) > > > At the same time, want to use another identifier as the basis for the > > > order. > > > > > > As the message key has many uses, avoid duplicate messages, compaction > > > topics. > > > I prefer provide a mechanism to overwrite key based sorting behavior. > > > Of course this is not mandatory. > > > > > > If the basis for sorting is exactly the same as the real purpose of the > > > message key, > > > just use the message key. Otherwise user provide the order key to > > > overwrite. > > > > > > Best Regards. > > > Penghui > > > > > > Sijie Guo <guosi...@gmail.com> 于2019年4月4日周四 下午3:36写道： > > > > > >> Jia thank you for raising this up. > > >> > > >> A couple of comments here > > >> > > >> 1. technically you don't need a consistent hashing mechanism for > > dividing > > >> hash ranges for consumers. Any hash mechanism that can map a key to a > > >> consumer should work here. The only advantage of consistent hashing > > >> mechanism is to reduce the disruptions when a consumer joins and > leaves. > > >> So > > >> I would prefer making the hashing mechanism pluggable. > > >> > > >> 2. regarding using key or introducing a new ordering key. I would > prefer > > >> using `key` here. having two `keys` actually makes people confused. > The > > >> `key` was used for compaction but the compaction doesn't mutate the > > >> original data. it just creates a new compacted ledger. in most of the > > >> time, > > >> when people use `key_failover`, they are probably accessing the > original > > >> data. Event they are trying to subscribe to compacted topic using > > >> `key_failover`, the ordering of messages by key will still be true. > so I > > >> wouldn't worry about this part that much. > > >> > > >> 3. in `failover` there is a consumer listener that listens on the > > >> partition > > >> assignment changes. Is there any requirements or plans on adding > similar > > >> listeners for `key_failover`? > > >> > > >> Thanks, > > >> Sijie > > >> > > >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zhai...@apache.org> wrote: > > >> > > >> > Hi all, > > >> > > > >> > I would like to init a discuss of "PIP 34: Add new subscribe type -- > > >> > Key_Failover" here. > > >> > > > >> > At present, there are 3 kinds of subscription modes: exclusive, > > shared, > > >> and > > >> > failover. Shared mode subscription is used a lot, because consumers > > >> could > > >> > effectively parallel consume messages of same partition. But using > > >> shared > > >> > mode will not keep the message order of same key, It would be good > to > > >> > leverage share mode, while also keeping the ordering of messages. > > >> > > > >> > This Proposal trying to introduce a new subscribe type Key_Failover > to > > >> > extend shared type. By Key_Failover, one partition could have > several > > >> > consumers to parallel consume messages, while all messages with the > > same > > >> > key will be dispatched to only one consumer. > > >> > > > >> > Here is the link contains more information of this PIP: > > >> > > > >> > > > >> > > > https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover > > >> > > > >> > Thanks. > > >> > > > >> > > > > > >
