Dave, Colin described the producer behavior of picking the partition for a message before it is sent to Kafka broker correctly. However, I'm interested in knowing your use case a little before to see why you would rather have the broker decide the partition?
Thanks, Neha On Tue, May 21, 2013 at 12:05 PM, Colin Blower <cblo...@barracuda.com>wrote: > The key is used by the client to decide which partition to send the > message to. By the time the client is creating the produce request, it > should be known which partition each message is being sent to. I believe > Neha described the behavior of the Java client which sends messages with > a null key to any partition. > > The key is described in past tense because of the use case for > persisting keys with messages. The key is persisted through the broker > so that a consumer knows what key was used to partition the message on > the producer side. > > I don't believe that you can have the broker decide which partition a > message goes to. > > -- > Colin B. > > On 05/21/2013 11:48 AM, Dave Peterson wrote: > > I'm looking at the document entitled "A Guide to the Kafka Protocol" > > located here: > > > > https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html > > > > It shows a produce request as containing a number of message sets, which > are > > grouped first by topic and second by partition (a 32-bit integer). > > However, each > > message in a message set contains a Key field, which is described as > follows: > > > > The key is an optional message key that was used for partition > assignment. > > The key can be null. > > > > I notice the use of "was" (past tense) above. That seems to suggest > that the > > Key field was once used to specify a partition (at the granularity of > each > > individual message), but the plan for the future is to instead use the > 32-bit > > partition value preceding each message set. Is this correct? If so, > when I am > > creating a produce request for 0.8, what should I use for the 32-bit > partition > > value, and how does this relate to the Key field of each individual > message? > > Ideally, I would like to just send a produce request and let the broker > choose > > the partition. How do I accomplish this in 0.8, and are there plans to > change > > this after 0.8? > > > > Thanks, > > Dave > > > > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > >> No. In 0.8, if you don't specify a key for a message, it is sent to any > of > >> the available partitions. In other words, the partition id is selected > on > >> the partition and the server doesn't get -1 as the partition id. > >> > >> Thanks, > >> Neha > >> > >> > >> On Tue, May 21, 2013 at 9:54 AM, Dave Peterson <dspeter...@tagged.com > >wrote: > >> > >>> In the version 0.8 wire format for a produce request, does a value of > -1 > >>> still indicate "use a random partition" as it did for 0.7? > >>> > >>> Thanks, > >>> Dave > >>> > > >
