Update:
I am using kafka.clients 0.8.2-beta. Below are the test steps
1. setup local kafka clusters with 2 brokers, 0 and 1
2. create topic X with replication fact 1 and 4 partitions
3. verify that each broker has two partitions
4. shutdown broker 1
5. start a producer sending data to topic X using KafkaProducer with
required ack = 1
6. producer hangs and does not exit.
Offline partitions were take care of when the partitions is null (code
attached below). However, the timeout setting does not seem to work. Not
sure what caused KafkaProducer to hang.
// choose the next available node in a round-robin fashion
for (int i = 0; i < numPartitions; i++) {
int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
if (partitions.get(partition).leader() != null)
return partition;
}
// no partitions are available, give a non-available partition
return Utils.abs(counter.getAndIncrement()) % numPartitions;
On Fri, Feb 20, 2015 at 1:48 PM, Xiaoyu Wang <xw...@rocketfuel.com> wrote:
> Hello,
>
> I am experimenting sending data to kafka using KafkaProducer and found
> that when a partition is completely offline, e.g. a topic with replication
> factor = 1 and some broker is down, KafkaProducer seems to be hanging
> forever. Not even exit with the timeout setting. Can you take a look?
>
> I checked code and found that the partitioner create partition based on
> the total partition number - including those offline partitions. Is it
> possible that we change ProducerClient to ignore offline partitions?
>
>
> Thanks,
>
> -Xiaoyu
>
>
