@Sijie Guo <si...@apache.org> Sorry, I may not have described my point in the previous email. I try to describe it again. :)
User can use key to achieve business logic(e.g. use user_id as message key) At the same time, want to use another identifier as the basis for the order. As the message key has many uses, avoid duplicate messages, compaction topics. I prefer provide a mechanism to overwrite key based sorting behavior. Of course this is not mandatory. If the basis for sorting is exactly the same as the real purpose of the message key, just use the message key. Otherwise user provide the order key to overwrite. Best Regards. Penghui Sijie Guo <guosi...@gmail.com> 于2019年4月4日周四 下午3:36写道： > Jia thank you for raising this up. > > A couple of comments here > > 1. technically you don't need a consistent hashing mechanism for dividing > hash ranges for consumers. Any hash mechanism that can map a key to a > consumer should work here. The only advantage of consistent hashing > mechanism is to reduce the disruptions when a consumer joins and leaves. So > I would prefer making the hashing mechanism pluggable. > > 2. regarding using key or introducing a new ordering key. I would prefer > using `key` here. having two `keys` actually makes people confused. The > `key` was used for compaction but the compaction doesn't mutate the > original data. it just creates a new compacted ledger. in most of the time, > when people use `key_failover`, they are probably accessing the original > data. Event they are trying to subscribe to compacted topic using > `key_failover`, the ordering of messages by key will still be true. so I > wouldn't worry about this part that much. > > 3. in `failover` there is a consumer listener that listens on the partition > assignment changes. Is there any requirements or plans on adding similar > listeners for `key_failover`? > > Thanks, > Sijie > > On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zhai...@apache.org> wrote: > > > Hi all, > > > > I would like to init a discuss of "PIP 34: Add new subscribe type -- > > Key_Failover" here. > > > > At present, there are 3 kinds of subscription modes: exclusive, shared, > and > > failover. Shared mode subscription is used a lot, because consumers could > > effectively parallel consume messages of same partition. But using > shared > > mode will not keep the message order of same key, It would be good to > > leverage share mode, while also keeping the ordering of messages. > > > > This Proposal trying to introduce a new subscribe type Key_Failover to > > extend shared type. By Key_Failover, one partition could have several > > consumers to parallel consume messages, while all messages with the same > > key will be dispatched to only one consumer. > > > > Here is the link contains more information of this PIP: > > > > > https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover > > > > Thanks. > > >
