Re: New and old producers partition messages differently

2015-04-26 Thread Gwen Shapira
Definitely +1 for advertising this in the docs. What I can't figure out is the upgrade path... if my application assumes that all data for a single user is in one partition (so it subscribes to a single partition and expects everything about a specific subset of users to be in that partition), thi

Re: New and old producers partition messages differently

2015-04-26 Thread Gwen Shapira
We are doing work for supporting custom partitioner, so everything is customizable :) On Sun, Apr 26, 2015 at 8:52 PM, Wes Chow wrote: > > Along these lines too, is the function customizable? I could see how mmh3 > (or 2) would be generally sufficient, however in some cases you may want > someth

Re: New and old producers partition messages differently

2015-04-26 Thread Wes Chow
Along these lines too, is the function customizable? I could see how mmh3 (or 2) would be generally sufficient, however in some cases you may want something that's a bit more cryptographically secure so as to avoid attacks. (Though I suppose the programmer could first crypto-hash the key, an

Re: New and old producers partition messages differently

2015-04-26 Thread Evan Huus
Related to this topic: why the choice of murmur2 over murmur3? I'm not super-familiar with the differences between the two, but I'd assume murmur3 would be faster or have a more even distribution or something. Evan P.S. Also, there appear to be many murmur3 implementations for other languages, wh

Re: New and old producers partition messages differently

2015-04-26 Thread Jay Kreps
This was actually intentional. The problem with relying on hashCode is that (1) it is often a very bad hash function, (2) it is not guaranteed to be consistent from run to run (i.e. if you restart the jvm the value of hashing the same key can change!), (3) it is not available outside the jvm so no

Re: New and old producers partition messages differently

2015-04-26 Thread Daniel Compton
I would support a configuration flag to be added in the short term, say until 0.9. In the long term, hashcode may change out from underneath people anyway, so delaying moving to Murmur for too long is likely to still end up in pain. Leaving that configuration around long term increases code and te

Re: Regarding key to b sent as part of producer message Please help

2015-04-26 Thread Gaurav Agarwal
Hi I got it was doing something wrong. need to set the value like this,topic is by default string KeyedMessage keyedMessage = new KeyedMessage("topic",Serializations.serialize("s"), Serializations.getSerialized(msg,rqst)); On Sun, Apr 26, 2015 at 3:14 AM, Manoj Khangaonkar wrote