Penguin, Thank you for your clarification! That makes sense to me then.
Sijie On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <codelipeng...@gmail.com> wrote: > @Sijie Guo <si...@apache.org> > > Sorry, I may not have described my point in the previous email. > I try to describe it again. :) > > User can use key to achieve business logic(e.g. use user_id as message key) > At the same time, want to use another identifier as the basis for the > order. > > As the message key has many uses, avoid duplicate messages, compaction > topics. > I prefer provide a mechanism to overwrite key based sorting behavior. > Of course this is not mandatory. > > If the basis for sorting is exactly the same as the real purpose of the > message key, > just use the message key. Otherwise user provide the order key to > overwrite. > > Best Regards. > Penghui > > Sijie Guo <guosi...@gmail.com> 于2019年4月4日周四 下午3:36写道： > >> Jia thank you for raising this up. >> >> A couple of comments here >> >> 1. technically you don't need a consistent hashing mechanism for dividing >> hash ranges for consumers. Any hash mechanism that can map a key to a >> consumer should work here. The only advantage of consistent hashing >> mechanism is to reduce the disruptions when a consumer joins and leaves. >> So >> I would prefer making the hashing mechanism pluggable. >> >> 2. regarding using key or introducing a new ordering key. I would prefer >> using `key` here. having two `keys` actually makes people confused. The >> `key` was used for compaction but the compaction doesn't mutate the >> original data. it just creates a new compacted ledger. in most of the >> time, >> when people use `key_failover`, they are probably accessing the original >> data. Event they are trying to subscribe to compacted topic using >> `key_failover`, the ordering of messages by key will still be true. so I >> wouldn't worry about this part that much. >> >> 3. in `failover` there is a consumer listener that listens on the >> partition >> assignment changes. Is there any requirements or plans on adding similar >> listeners for `key_failover`? >> >> Thanks, >> Sijie >> >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zhai...@apache.org> wrote: >> >> > Hi all, >> > >> > I would like to init a discuss of "PIP 34: Add new subscribe type -- >> > Key_Failover" here. >> > >> > At present, there are 3 kinds of subscription modes: exclusive, shared, >> and >> > failover. Shared mode subscription is used a lot, because consumers >> could >> > effectively parallel consume messages of same partition. But using >> shared >> > mode will not keep the message order of same key, It would be good to >> > leverage share mode, while also keeping the ordering of messages. >> > >> > This Proposal trying to introduce a new subscribe type Key_Failover to >> > extend shared type. By Key_Failover, one partition could have several >> > consumers to parallel consume messages, while all messages with the same >> > key will be dispatched to only one consumer. >> > >> > Here is the link contains more information of this PIP: >> > >> > >> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover >> > >> > Thanks. >> > >> >
