>When the producer
>tries to send to the old broker (which is either dead, or a slave now),
>the broker will either not respond, or the response will contain an error
>code.
In this case, the broker sends a response with an error code to the
producer, and then the producer retries the metadata r
Hey Guys,
Correct me if I'm wrong, but I believe in 0.8, the producer uses a
metadata request to get topic/partition mappings from the broker. The
broker then interacts with ZK (rather than having the producer do it using
zk.connect).
In the event that the master for a topic/partition fails, a ne
> Can I guess that the auto detecting ability for producer is weakened in
> Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?
In 0.8, we got rid of zookeeper from the producer and replaced its
functionality with a metadata API. On startup and whenever a request fails,
the produc
That's an experiment for a future release. The consumer logic in 0.8 is
more or less the same as 0.7, ie, running in a decentralized way.
Thanks,
Jun
On Tue, Jan 15, 2013 at 12:10 AM, gj1989lh wrote:
> Hi,
> How is the consumer rebalancing works?
> In the design part of the official document,
All apis for 0.8 are documented in
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
However, there is no need for a producer to call the metadata api directly.
Our producer client does this for you automatically.
For your first question, if a new broker is added, cu
Hi,
How is the consumer rebalancing works?
In the design part of the official document, it says each consumer decides
which partitions it should own. But in below document
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination
It says there will be a centralized consumer c
Hi,
We know in Kafka 0.7, we can specify zk.connect. And with zookeeper, the
producer can dynamically detect broker. But in Kafka 0.8, we can't specify
zk.connect for producer. How does the producer in Kafka 0.8 auto detect broker?
I have done two experiments.
In first one, I configure the broke
Thanks for your kindly reply.
From: Jun Rao [mailto:jun...@gmail.com]
Sent: 2013年1月15日 13:53
To: dev@kafka.apache.org; Jun Guo -X (jungu - CIIC at Cisco)
Subject: Re: About kafka 0.8 producer zookeeper-based load balancing on
per-request basis
Basically, we spread partitions among multiple
@kafka.apache.org
Cc: Jun Guo -X (jungu - CIIC at Cisco)
Subject: Re: About kafka 0.8 producer zookeeper-based load balancing on
per-request basis
Than how it achieve zookeeper-based load balance on per-request basis?
You are asking two related but different questions. One is how does load
balancing work
>
>
> Than how it achieve zookeeper-based load balance on per-request basis?
You are asking two related but different questions. One is how does load
balancing work and the other is how does broker discovery works. Jun
explained how load balancing works and how requests are routed to
partitons. I
Basically, we spread partitions among multiple brokers. If a message is
sent without a key, the producer picks a random partition to balance the
load. If a message has a key, the default partitioner hashes the key to one
of the partitions deterministically. Then, the load may not always be
balanced
Yes, in neither 0.7 or 0.8 should you need to change either the producer,
consumer, or broker config when adding a new broker.
-jay
On Wed, Jan 9, 2013 at 8:18 PM, 杨涛 wrote:
> Hi,
>
> Then is to say: for kafka 0.8, when you add a new broker into the
> cluster, you need not to change t
Hi,
Then is to say: for kafka 0.8, when you add a new broker into the
cluster, you need not to change the producer broker setting, is it right?
Bruce
In other words 0.8 gives you the richer functionality of the zk producer
without requiring a zk connection. As with zk, you need to give it one or
more urls to bootstrap a connection, but these urls don't need to be the
complete set of brokers (just choose a couple in case one is down or set up
a v
In 0.8, the producer load balances data across the Kafka brokers using a
special getMetadata API. You still need to wire in a broker.list since the
producer will use this list to talk to one of the brokers to get the
leadership information for a new topic/partition. If the leader moves over
time, t
Hi all,
I find that in Kafka 0.72, we can specify either zk.connect or
broker.list. But in Kafka 0.8, we can only specify broker.list ,and we can't
specify zk.connect without specifying broker.list. I think, in this case, we
can't balance producer through zookeeper. If anyone use Kafka 0.
16 matches
Mail list logo
