Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is
being used to actually send messages through. We're seeing this behavior after
a restart and deletion of the kafka data (/tmp/kafka-logs). I
For some extra info, this was a restart of the just zookeeper and kafka, our
clients were all still online.
Karl
On Apr 23, 2013, at 10:50 AM, Karl Kirch
wrote:
> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
> per node) and Zookeeper 3.3.4 on each nod
Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
per node) and Zookeeper 3.3.4 on each node. When we first start up the cluster,
there are no issues and things work just fine, but after a while things start
acting a bit haywire. It looks like we start to miss messages
Zookeeper based.
Karl
On Apr 23, 2013, at 1:15 PM, Neha Narkhede
wrote:
> What kind of producer do you use ? Zookeeper based or broker list ?
>
> Thanks,
> Neha
>
> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch wrote:
>> For some extra info, this was a restart o
running to
> https://issues.apache.org/jira/browse/KAFKA-278 ?
>
> Thanks,
> Neha
>
> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch wrote:
>> Zookeeper based.
>>
>> Karl
>>
>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede
>> wrote:
>>
>>&
arting up all of the kafka brokers and then the zookeeper
instances. I've also tried bringing up each machine (zookeeper and kafka) up
individually. All seem to give inconsistent results.
Is there a recommended way to bring nodes online?
Karl
On Apr 23, 2013, at 1:36 PM, Karl Kirch wrote:
point?
Karl
On Apr 23, 2013, at 2:11 PM, Xavier Stevens
wrote:
> You should bring up your Zookeeper instances first and then the Kafka
> brokers.
>
>
> On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch wrote:
>
>> Now to make things even more interesting. I restarted 2 a
I occasionally am getting some batch send errors from the stock async producer.
This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
Is there anyway to check what happens when those batch errors occur?
Or bump up the retry count? (looks like it only did a single retry).
I need the spe
ast get a bit more of a chance to recover.
Thanks,
Karl
On Apr 23, 2013, at 5:35 PM, Karl Kirch
wrote:
> I occasionally am getting some batch send errors from the stock async
> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> Is there anyway to check wha
Hmmm… that didn't seem to help.
Anyone else see this sort of errors?
Karl
On Apr 23, 2013, at 5:58 PM, Karl Kirch
wrote:
> I'm going to try bumping up the "numRetries" key in my producer config.
> Is this a good option in this case?
> I am using the zookeepe
failing when
> the batch is submitted.
>
>
> On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch wrote:
>
>> Hmmm… that didn't seem to help.
>> Anyone else see this sort of errors?
>>
>> Karl
>>
>>
>> On Apr 23, 2013, at 5:58 PM, Karl
that out?
>
> Thanks,
>
> Jun
>
>
> On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch wrote:
>
>> I occasionally am getting some batch send errors from the stock async
>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>> Is there anyway
you are producing faster than the
> brokers can persist the messages. It's possible these are two different
> issues...
>
> Anyway I've been doing a lot of work on this this afternoon so I may have
> more information later. Someone else probably knows more though.
>
&
So I'm seeing CancelledKeyExceptions cropping up about the time that the
connections get reset.
Is this a zookeeper error that I'm hitting?
Karl
On Apr 24, 2013, at 9:55 AM, Karl Kirch
wrote:
> Just got logging cranked up. Will let you know when I see it again.
>
> Tha
you think they are.
>
> Have you tried using the sync producer to work out your bugs? My guess is
> the sync producer would fail on the first message rather than failing when
> the batch is submitted.
>
>
> On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch wrote:
>
>> H
Any advice on using a static broker list vs using zookeeper? I keep having fits
with keeping things stable with zookeeper involved (i.e. dropped connections).
If I use a static broker list do I still get failover if a broker goes down?
(i.e. 1 broker goes down, will my producers still try to sen
Is there a best practice on how to handle producer objects for long running
apps?
Right now I have an app that is long running and will sit for large stretches
of time (days/weeks) with next to none load and then get slammed. In that case
I'd like to cache the producer so I don't incur a hit i
17 matches
Mail list logo
