Re: ConsumerConnector connecting to Zookeeper: EndOfStreamException

Could you try setting a larger zookeeper.connection.timeout.ms? Thanks, Jun On Tue, May 6, 2014 at 4:16 AM, Sebastian Mattheis < sebastian.matth...@bmw-carit.de> wrote: > Hello, > > I get an error when trying to 'consume' messages from Kafka (2.9.2-0.8.1) > with a Zookeer stand-alone (3.4.5).

Re: owner info in zk is not correct

Thanks GuoZhang. After last accident, we stop all consumers and then restart all consumers one by one then it is ok. 2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream So consumer1 consumes 10 partitions, consumer2 consumers another 10 partitions, consumer3 is idle. Tod

SimpleConsumer long poll test case randomly times out

I've got a very strange bug doing a long poll using SimpleConsumer on 0.8.1.1. The test app I've got (attached) uses a thread to do a long poll of a topic, and then sends single messages to that queue, one at a time, waiting for the poll to return the message before sending the next one. The pr

Re: question about isr

This seems similar to behavior we’re seeing. At some point one of our brokers (id 1) just gives up and starts throwing those errors and kafka-topics no longer lists it as a ISR. However the logs for that broker say something very odd: [2014-05-09 10:16:00,248] INFO Partition [callbackServiceTop

Re: Controller to broker connection timing out through load balancer

0.8.1.1 On Tue, May 13, 2014 at 9:02 PM, Jun Rao wrote: > Which version of Kafka are you using? > > Thanks, > > Jun > > > On Tue, May 13, 2014 at 5:56 PM, Robin Yamaguchi >wrote: > > > It seems like this mailing list wasn't updating through the web archives > > for a few days last week, so I w

Re: What happens to Kafka when ZK lost its quorum?

Kafka requires a functional and healthy zookeeper setup. It is recommended that you closely monitor your zookeeper cluster and provision it so that it is performant. Thanks, Neha On Tue, May 13, 2014 at 6:52 AM, Connie Yang wrote: > Hi all, > > Can Kafka producers, brokers and consumers still

how to know kafka producer api status

If l use java producer api in sync mode. public void send(kafka.producer.KeyedMessage message) { /* compiled code */ } How to know whether a send process is successful or failed? For example if the kafka broker disk is not accessible , will it throw exceptions?

Re: JAVA HEAP settings for KAFKA in production

Just curious if these JAVA_HEAP tunings are still for machines listed in 6.4 Hardware and OS : "We are using dual quad-core Intel Xeon machines with 24GB of memory." Seems like a small heap size for 24GB of memory, or is that done to maximize as

Re: What happens to Kafka when ZK lost its quorum?

Hi Guozhang, In worst case zookeeper dies for say 1 hour and come back up, will things still be recovered automatically after 1 hour ? Weide On Wed, May 14, 2014 at 8:28 AM, Guozhang Wang wrote: > In 0.8, the servers and consumers are heavily dependent on ZK to function. > With ZK down, the s

Re: Controller to broker connection timing out through load balancer

Any error in the controller and state-change log? Thanks, Jun On Tue, May 13, 2014 at 9:16 PM, Robin Yamaguchi wrote: > 0.8.1.1 > > > On Tue, May 13, 2014 at 9:02 PM, Jun Rao wrote: > > > Which version of Kafka are you using? > > > > Thanks, > > > > Jun > > > > > > On Tue, May 13, 2014 at 5:5

Re: kafka performance question

How many brokers and partitions do you have? You may try increasing batch.num.messages. Thanks, Jun On Tue, May 13, 2014 at 5:56 PM, Zhujie (zhujie, Smartcare) < first.zhu...@huawei.com> wrote: > Dear all, > > We want to use kafka to collect and dispatch data file, but the > performance is may

Re: Controller to broker connection timing out through load balancer

It seems like this mailing list wasn't updating through the web archives for a few days last week, so I wanted to send this out again in case it wasn't seen. My apologies for the repost. In further troubleshooting, I've also observed if a broker is shut down while a connection in is CLOSE_WAIT, t

How To Use kafka-topics.sh To Move Replica To Different Broker

Hi Guys, I’m using Kafka 0.8.1, and I’m having difficulty trying to move a replica from one broker to another. Doing a "kafka-topics.sh —describe | grep mini__078active_80__32__mini" shows: Topic:mini__078active_80__32__mini PartitionCount:1 ReplicationFactor:3 Configs:retention.byte

Finding out how underreplicated a broker is

In the output of kafka-topics.sh -describe, it gives the ISR. What I want to find out is why a particular broker is not in ISR, how far behind it is, and when it will get back into the ISR. I can't seem to find this information in the logs, cmdline output, or zookeeper. How can I find out how fa

Weird behaviour of Kafka consumer/mirrormaker at different DCs

Hi , I have been setting up 2 clusters of Kafka located across DCs , one in Asia and another one in Europe. All of them are running on EC2 m1.xlarge machines. Our goal is to replicate data from Asia cluster to Europe cluster with "high and stable" speed. I ended up trying Mirrormaker provided b

Re: Weird behaviour of Kafka consumer/mirrormaker at different DCs

To amortize the long latency across DC, you may need to tune the socket buffer size to get higher throughput. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 Thanks, Jun On Fri, May 9, 2014 at 4:06 AM, Sithiyavanich, Manawat (Agoda) < manawat.sithiyavan...@agoda.co

Leader for almost all topic, partitions on the same broker

Hello, I have a 3-node cluster that has had a couple issues lately. One thing I'm trying to sort out is why the topic-partitions are all owned by a single leader (when i list topics, the leader is assigned to the current controller node, irrespective the preferred replica). I have tried to use t