Xiaoyu,
For 1, I have a patch for 0.8.2 in
https://issues.apache.org/jira/browse/KAFKA-1984. Could you test it out and
see if it fixes your issue?
For 2, I did some local testing. The only issue I saw is that producer can
block on close since there are still unsent messages in the bufferpool.
Thi
Jun,
Can you also take a look at the second problem I am having?
> > > I am trying to test how KafkaProducer behaves with topic replication
> > factor
> > > = 1
> > >
> > >1. One broker is offline BEFORE KafkaProducer starts sending
> messages.
> > >Because of the bug I mentioned, KafkaP
Ah, yes. You are right. That's a more obvious bug. Will fix that in
KAFKA-1984.
Thanks,
Jun
On Tue, Feb 24, 2015 at 8:37 AM, Xiaoyu Wang wrote:
> Hi Jun,
>
> If I understand it correctly. the highlighted line is for avoiding
> offline partitions, is it?
>
> for (int i = 0; i < numPartitions; i
Hi Jun,
If I understand it correctly. the highlighted line is for avoiding
offline partitions, is it?
for (int i = 0; i < numPartitions; i++) {
int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
if (partitions.get(partition).leader() != null) {
return partition;
Hi, Xiaoyu,
1. Could you explain a bit more what the bug is? The code does try to avoid
picking an unavailable partition. There does seem to be an issue when there
are more than one thread producing data to the same producer instance. This
is being tracked in KAFKA-1984. How many producing threads
Jun,
I am trying to test how KafkaProducer behaves with topic replication factor
= 1
1. One broker is offline BEFORE KafkaProducer starts sending messages.
Because of the bug I mentioned, KafkaProducer sends to the offline
partition and hangs forever.
2. One broker goes offline WHILE
The logic in that code is to cycle through all partitions and return as
soon as we see a partition with the leader. I do see an issue that if there
are multiple threads sending messages to the same producer concurrently, we
may not cycle through all partitions and therefore we could return an
unava
Found the problem - it is a bug with Partitions of kafka client. Can you
guys confirm and patch in kafka clients?
for (int i = 0; i < numPartitions; i++) {
int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
if (partitions.get(partition).leader() != null) {
return
Update:
I am using kafka.clients 0.8.2-beta. Below are the test steps
1. setup local kafka clusters with 2 brokers, 0 and 1
2. create topic X with replication fact 1 and 4 partitions
3. verify that each broker has two partitions
4. shutdown broker 1
5. start a producer sending dat
Hello,
I am experimenting sending data to kafka using KafkaProducer and found that
when a partition is completely offline, e.g. a topic with replication
factor = 1 and some broker is down, KafkaProducer seems to be hanging
forever. Not even exit with the timeout setting. Can you take a look?
I ch
10 matches
Mail list logo
