Yes I believe the TopicErrorCode is also an int16. I will update the
documentation accordingly.
In general, if the protocol specification is not explicit it should be
made explicit. This will help other client driver authors in the future.
If you find any other errors or omissions in the documenta
Yes. You can see the details of the wire protocol here
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataResponse
Thanks,
Neha
On May 23, 2013 9:43 AM, "Dave Peterson" wrote:
> Ok, thanks for the information. Looking at the wire for
Ok, thanks for the information. Looking at the wire format for the
metadata response, I see that the right hand side of the TopicMetadata
production contains a TopicErrorCode, and the right hand side of the
PartitionMetadata production contains a PartitionErrorCode. Are both
of these 16-bit value
1. Correct
2. The producer does not use or depend on zookeeper anymore. It refreshes
its view of the cluster metadata by using a TopicMetadataRequest to any of
the kafka brokers. It maps a message to a partition using the following
rules -
2.1 If a message has no key, use any available partition
2.
Ok, the picture I have in my mind of how things work in 0.8 (from a
producer's point of view) is as follows:
1. An application program sends log messages to a producer. Each
message is provided as a key/value pair, where the key is chosen
by the application and the value is t
Dave,
Currently, the broker expects each producer request to specify the exact
partition id (-1 is on longer valid). The mapping from a message to a
partition is done at the producer client. The producer can choose a random
partition (from the existing list of partitions) or deterministically
choo
In my case, there is a load balancer between the producers and the
brokers, so I want the behavior described for the Java client (null key
specifies "any partition"). If the Key field of each individual message
specifies the partition to send it to, then I don't understand the purpose
of the 32-bi
Dave,
Colin described the producer behavior of picking the partition for a
message before it is sent to Kafka broker correctly. However, I'm
interested in knowing your use case a little before to see why you would
rather have the broker decide the partition?
Thanks,
Neha
On Tue, May 21, 2013 at
The key is used by the client to decide which partition to send the
message to. By the time the client is creating the produce request, it
should be known which partition each message is being sent to. I believe
Neha described the behavior of the Java client which sends messages with
a null key to
I'm looking at the document entitled "A Guide to the Kafka Protocol"
located here:
https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html
It shows a produce request as containing a number of message sets, which are
grouped first by topic and second by partition (a 32-bit integer).
No. In 0.8, if you don't specify a key for a message, it is sent to any of
the available partitions. In other words, the partition id is selected on
the partition and the server doesn't get -1 as the partition id.
Thanks,
Neha
On Tue, May 21, 2013 at 9:54 AM, Dave Peterson wrote:
> In the versi
In the version 0.8 wire format for a produce request, does a value of -1
still indicate "use a random partition" as it did for 0.7?
Thanks,
Dave
12 matches
Mail list logo
