Hi, All
We encounter an issue with Kafka of messages loss:
We have Kafka 0.7.0 installed in cluster mode on three nodes(on partition
per node configured).
We are using kafka java API to send bulk of messages(1000 messages in each)
in sync mode.
After the last bulk is sent we close the producer.
On 12-11-28 09:54 PM, Jun Rao wrote:
Dmitri,
Could you reproduce this easily? Are you using a load balancer? Earlier,
another user had the same issue and eventually figured out that the problem
is in the network router.
I do not believe I have any loadbalancer anywhere in the picture and by
the
log.flush.interval is set to 1. Should I increase or decrease?
thanks
Jamie
-Original Message-
From: Jun Rao [mailto:jun...@gmail.com]
Sent: Wednesday, November 28, 2012 10:00 PM
To: users@kafka.apache.org
Subject: Re: producer queue size
Is the server busy on I/O? What's log.flush
Hi Boris,
In Kafka 0.7, the producer does not get any ACK from the server, so
the protocol is fire and forget. What could happen is if a broker is
shutting down, the messages might still live in the producer's socket
buffer and the server can shutdown before getting a chance to read the
producer's
testing new list
On Wed, Nov 28, 2012 at 10:44 AM, Jun Rao wrote:
> You can find the information at
> http://incubator.apache.org/kafka/design.html
>
> Look for consumer registration algorithm and consumer rebalancing
> algorithm.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 28, 2012 at 7:13 AM, S A
Re-sending since I didn't get any responses before and want to make sure this
made it to the correct group.
From: Sybrandy, Casey
Sent: Wednesday, November 28, 2012 2:39 PM
To: users@kafka.apache.org
Subject: Zookeeper Configuration Question
All,
We're having an
My consumers randomly hang. They essentially end up waiting on a kafka
stream:
There is plenty of data on the brokers and a ratio of about 1 consumer for
5 partitions.
Any thoughts on how to keep the stream going? Or at least restart on hang?
e.g.
ConsumerConfig consumerConfig =
-- Forwarded message --
From: Juan Valencia
Date: Thu, Nov 29, 2012 at 11:45 AM
Subject: Consumer Hanging (yes, read the FAQ)
To: users@kafka.apache.org
My consumers randomly hang. They essentially end up waiting on a kafka
stream:
There is plenty of data on the brokers and a r
A popular case is that the thread isn't catching thrown Exceptions in run()
- which causes the thread to be restarted in a typical Executor
configuration - and does not pull any more items.
Is that the case?
On Thu, Nov 29, 2012 at 11:48 AM, Juan Valencia wrote:
> -- Forwarded message -
On Thu, Nov 29, 2012 at 10:46 AM, Sybrandy, Casey
wrote:
> Re-sending since I didn't get any responses before and want to make
> sure this made it to the correct group.
>>We're having an issue with Zookeeper, which has nothing to do with
>>Kafka, but my consumers don't appear to be attempting to
On Thu, Nov 29, 2012 at 1:15 PM, James A. Robinson
wrote:
> For my kafka startup I point to the zookeeper "cluster" like so:
>
> --kafka-zk-connect
> logproc-dev-03:2181,logproc-dev-03:2182,logproc-dev-03:2183
Sorry, wrong copy and paste! For the kafka startup I point to the zookeeper
cluster
The thread dump basically shows that the consumer thread (from your
executor) is waiting on the internal fetcher queue. This means the
internal fetcher queue is empty, which in turn suggests that the
fetcher might not be running.
Please can you file a bug and share the entire thread dump there ?
T
Please can you send around the log that shows the zookeeper connection
error ? I would like to see if it fails at connection establishment or
session establishment.
Thanks,
Neha
On Thu, Nov 29, 2012 at 1:19 PM, James A. Robinson
wrote:
> On Thu, Nov 29, 2012 at 1:15 PM, James A. Robinson
> wrot
Dmitri,
You said that the problem goes away once you upgrade to 0.7.1. Do you
mean that the message corruption doesn't happen or that when it
happens, the broker doesn't shut down ?
I'm asking since we occasionally do see the corruption on 0.7.1 but
don't have a good way to reproduce that. 0.7.1 f
On 11/29/2012 01:50 PM, Neha Narkhede wrote:
> Dmitri,
>
> You said that the problem goes away once you upgrade to 0.7.1. Do you
> mean that the message corruption doesn't happen or that when it
> happens, the broker doesn't shut down ?
I do not see corrupt message at all.
> I'm asking since we o
That just meant that we couldn't reproduce it during our testing, but
we occasionally do see it in production, which is much harder to
reproduce.
It will be great if you can reproduce the issue and attach the test case.
Thanks,
Neha
On Thu, Nov 29, 2012 at 2:03 PM, Dmitri Priimak
wrote:
> On 11/
On 11/29/2012 02:06 PM, Neha Narkhede wrote:
> That just meant that we couldn't reproduce it during our testing, but
> we occasionally do see it in production, which is much harder to
> reproduce.
> It will be great if you can reproduce the issue and attach the test case.
I see. I will try to isola
Hi All,
I am new to Kafka and we are planning to use Kafka as message broker.
I am wondering if there are java API documentation for Kafka. I noticed that
current API doc is written for Scala user. I believe it could be because it is
written
in Scala.
Please let me know.
Thanks
Subhash A.
Are you referring to the scaladoc?
http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/ There's
(some) documentation on the java api as well (i.e., the javaapi packages)
that you can use from Java.
Joel
On Thu, Nov 29, 2012 at 3:14 PM, Subhash Agrawal wrote:
> Hi All,
>
> I am new to
I think what Subhash is asking for is just plain old fashioned javadoc.
People are used to having that for java apis. We should do that, but I
think it would require moving the java-friendly wrappers into java. This
would probably be a good thing to do anyway just to get rid of all the
crazy scala
Hmm, that should be big enough.
Thanks,
Jun
On Thu, Nov 29, 2012 at 10:03 AM, Jamie Wang wrote:
> log.flush.interval is set to 1. Should I increase or decrease?
>
> thanks
> Jamie
>
> -Original Message-
> From: Jun Rao [mailto:jun...@gmail.com]
> Sent: Wednesday, November 28, 2012
21 matches
Mail list logo
