Yes, it is *very* misleading in my opinion - I've seen so many people surprised with that behaviour... Technically it's 100% correct of course: "If the key is null, then the Producer will assign the message to a random Partition." - that's what actually happens, because assignment is random. However, everyone I know (including myself) assumed that "random" means "random for each message", rather than "picked randomly every topic.metadata.refresh.ms". So no, you're not the only one who got that wrong ;-)
M. Kind regards, Michał Michalski, michal.michal...@boxever.com On 5 December 2014 at 18:32, Yury Ruchin <yuri.ruc...@gmail.com> wrote: > Hello, > > I've come across a (seemingly) strange situation when my Kafka producer > gave so uneven distribution across partitions. I found that I used null key > to produce messages, guided by the following clause in the documentation: > "If the key is null, then a random broker partition is picked." However, > after looking at the code, I found that the broker partition is not truly > random for every message - instead, the randomly picked partition number > sticks and only refreshes after the topic.metadata.refresh.ms expires, > which is 10 minutes by default. So, with null key the producer keeps > writing to the same partition for 10 minutes. > > Is my understanding of partitioning with null key correct? If yes, > shouldn't the documentation be fixed then to explicitly describe the sticky > pseudo-random partition assignment? > > Thanks, > Yury >
