no zk connection timeout in consumer config of kafka 0.7?

Hi, In http://kafka.apache.org/08/configuration.html, there are 2 parameter about zk in consumer. zookeeper.session.timeout.ms 6000 Zookeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur. zookeeper.con

first message is lost

Hi, For a non-existent topic, the consumer and producer are set up. Then if the producer sends the first message, producer gets this exception: [2014-11-12 16:24:28,041] WARN Error while fetching metadata [{TopicMetadata for topic test5 -> No partition metadata for topic test5 due to kafka.commo

Re: first message is lost

the best way to resolve this issue. > > Guozhang > > On Wed, Nov 12, 2014 at 12:35 AM, Yonghui Zhao > wrote: > > > Hi, > > > > For a non-existent topic, the consumer and producer are set up. > > Then if the producer sends the first message, producer g

kafka deleted old logs but not released

Hi, We use kafka_2.10-0.8.1.1 in our server. Today we found disk space alert. We find many kafka data files are deleted, but still opened by kafka. such as: _yellowpageV2-0/68170670.log (deleted) java 8446 root 724u REG 253,2 536937911 26087362 /home

Re: kafka deleted old logs but not released

> Jun > > On Tue, Jan 6, 2015 at 4:46 AM, Yonghui Zhao > wrote: > > > Hi, > > > > We use kafka_2.10-0.8.1.1 in our server. Today we found disk space alert. > > > > We find many kafka data files are deleted, but still opened by kafka. > > > >

Re: kafka deleted old logs but not released

CentOS release 6.3 (Final) 2015-01-07 22:18 GMT+08:00 Harsha : > Yonghui, >Which OS you are running. > -Harsha > > On Wed, Jan 7, 2015, at 01:38 AM, Yonghui Zhao wrote: > > Yes and I found the reason rename in deletion is failed. > > In rename progres

kafka shutdown automatically

Hi， our kafka cluster is shut down automatically today, here is the log. I don't find any error log. Anything wrong? [2015-01-18 05:01:01,788] INFO [BrokerChangeListener on Controller 0]: Broker change listener fired for path /brokers/ids with children 0 (kafka.controller.ReplicaStateMachine$B

Re: kafka shutdown automatically

0.8.1.1 we have 2 brokers. broker-0 and broker-1 2015-01-20 8:43 GMT+08:00 Guozhang Wang : > Yonghui, which version of Kafka are you using? And does your cluster only > have one (broker-0) server? > > Guozhang > > On Sat, Jan 17, 2015 at 11:53 PM, Yonghui Zhao > wrote:

Re: kafka deleted old logs but not released

? Is there something else that accesses or maybe deleting >> these log files? For more context to this question, please read the >> discussion related to this here http://mail-archives.apache. >> org/mod_mbox/kafka-dev/201501.mbox/%3C54C47E9B.5060401%40gmail.com%3E >> >>

Re: partition and consumer stream number

Thanks Jun, I didn't get you previous answer. I think I didn't state my question clearly. I know the number of the consumer in one consumer group should be less or equal than partition number. But my question is about the relation between the stream number in one consumer and partition number su

Re: Proper use of ConsumerConnector

In our project we use senseidb to consume kafka data. Senseidb will process the message immediately but won't flush to disk immeidately. So if senseidb crash then all result not flushed will be lost， we want to rewind kafka. The offset we want to rewind to is the flush checkpoint. In this case,

Re: java.io.IOException: Broken pipe

mer could've closed the socket due to some error or could've died. Thanks, Neha - Hide quoted text - <http://grokbase.com/t/kafka/users/133fwd3r91/java-io-ioexception-broken-pipe#> On Friday, March 15, 2013, Yonghui Zhao wrote: This exception happened many times in high

Re: Connection reset by peer

nnection reset" that we > have seen are caused by the load balancer or the firewall killing idle > connections. > > Thanks, > > Jun > > On Sun, Mar 17, 2013 at 8:24 PM, Yonghui Zhao > wrote: > > > In kafka 0.7.2, I use a producer to send 200 million message t

Re: Connection reset by peer

kafka.network.Processor.read(SocketServer.scala:311) at kafka.network.Processor.run(SocketServer.scala:214) at java.lang.Thread.run(Thread.java:679) 2013/3/19 Yonghui Zhao > Thanks Jun. > > Now I use onebox to test kafka, kafka server ip on zk is 127.0.0.1, > network is not affected by exte

Re: java.io.IOException: Broken pipe

db to see what the issue is. > > Thanks, > Neha > > On Monday, March 18, 2013, Yonghui Zhao wrote: > > > Thanks Neha, > > > > I use one kafka server with 4 partitions and 3 consumers(senseidb). > > > > Kafka server producer input rate is about 10k. &g

Re: Connection reset by peer

hen a produce request fails? > > Thanks, > > Jun > > On Tue, Mar 19, 2013 at 1:34 AM, Yonghui Zhao > wrote: > > > Connection reset exception reproed. > > > > [2013-03-19 16:30:45,814] INFO Closing socket connection to /127.0.0.1. > > (kafka.network.Processor

Re: Connection reset by peer

e thread was interrupted and therefore the > socket channel was closed. Are you using SimpleConsumer? > > Thanks, > > Jun > > On Wed, Mar 20, 2013 at 9:25 PM, Yonghui Zhao > wrote: > > > Hi Jun, > > > > I didn't find any error in producer log. > &

Re: Connection reset by peer

> Thanks, > > Jun > > On Thu, Mar 21, 2013 at 8:38 PM, Yonghui Zhao > wrote: > > > No, I use java consumer connector, and set 10 seconds timeout. > > > >ConsumerConfig consumerConfig = new ConsumerConfig(props); > > _consumerConnector = > &

Re: Connection reset by peer

gt; On Thu, Mar 21, 2013 at 9:40 PM, Yonghui Zhao > wrote: > > > The application won't shut down the consumer connector. The consumer is > > always alive. > > > > 2013/3/22 Jun Rao > > > > > If you use the high level consumer, normally ClosedBy

Re: Connection reset by peer

Hi Jun： We use 1 consumer 1 kafka server with 4 partitions of only 1 topic. 2013/3/22 Yonghui Zhao > Yes, before consumer exception: > > 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] [] > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing > consumer*

Re: Connection reset by peer

many rebalances can slow down the > consumption and you will need to tune your GC setting. > > Thanks, > > Jun > > On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao wrote: > >> Yes, before consumer exception: >> >> 2013/03/21 12:07:17.909 INFO [ZookeeperCo

commitOffsets java.lang.NullPointerException

Have you ever seen this exception? 2013/03/25 12:08:32.020 WARN [ZookeeperConsumerConnector] [] 0_lu-ml-test10.bj-1364184411339-7c88f710 exception during commitOffsets java.lang.NullPointerException at org.I0Itec.zkclient.ZkConnection.writeData(ZkConnection.java:111) at org.I0Itec

Re: commitOffsets java.lang.NullPointerException

ed or the client is > being shutdown. Do you mind filing a bug ? > > Thanks, > Neha > > On Sunday, March 24, 2013, Yonghui Zhao wrote: > > > Have you ever seen this exception? > > > > > > 2013/03/25 12:08:32.020 WARN [ZookeeperConsumerConnector] [] > &g

Re: commitOffsets java.lang.NullPointerException

Done, https://issues.apache.org/jira/browse/KAFKA-824 2013/3/25 Neha Narkhede > On the Kafka JIRA - https://issues.apache.org/jira/browse/KAFKA > > Thanks, > Neha > > On Sunday, March 24, 2013, Yonghui Zhao wrote: > > > Sure. no problem. Where should I file the

Re: Connection reset by peer

logs in the consume saying sth like "expired session" for ZK. > Occasional rebalances are fine. Too many rebalances can slow down the > consumption and you will need to tune your GC setting. > > Thanks, > > Jun > > On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao

Re: Connection reset by peer

Any suggestion on consumer side? 在 2013-3-25 下午9:49，"Neha Narkhede" 写道： > For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new gen > 256 MB, CMS collector with occupancy of 70%. > > Thanks, > Neha > > On Sunday, March 24, 2013, Yonghui Zhao wrot

Re: Connection reset by peer

get away with using CMS for the tenured generation and parallel > collector for the new generation with a small heap like 1gb or so. > > Thanks, > Neha > > On Monday, March 25, 2013, Yonghui Zhao wrote: > > > Any suggestion on consumer side? > > 在 2013-3-25 下午9:49，

Re: Connection reset by peer

kafka server is started by bin/kafka-server-start.sh. No gc setting. 在 2013-3-26 下午11:40，"Neha Narkhede" 写道： > Did you have a gc pause around that time on the server ? What are your > server's current gc settings ? > > Thanks, > Neha > > On Mon, Mar 25, 20

Re: Connection reset by peer

bably doing that is a > good idea. > > Thanks, > Neha > > On Tue, Mar 26, 2013 at 9:47 AM, Yonghui Zhao > wrote: > > kafka server is started by bin/kafka-server-start.sh. No gc setting. > > 在 2013-3-26 下午11:40，"Neha Narkhede" 写道： > > > >> D

Re: Connection reset by peer

the root cause. So, you will > need to tune the GC setting further. Another way to avoid ZK session > timeout is to increase the session timeout config. > > Thanks, > > Jun > > On Wed, Mar 27, 2013 at 8:35 PM, Yonghui Zhao > wrote: > > > Now I used GC like thi

Re: Connection reset by peer

g and see > how long and how frequent your GCs are. > > Thanks, > > Jun > > On Thu, Mar 28, 2013 at 12:23 AM, Yonghui Zhao >wrote: > > > I used zookeeper-3.3.4 in kafka. > > > > Default tickTime is 3 seconds, minSesstionTimeOut is 6 seconds. > > N

Re: Connection reset by peer

r as well. > > Thanks, > > Jun > > On Thu, Mar 28, 2013 at 8:20 AM, Yonghui Zhao > wrote: > > > Thanks Jun. > > > > But I can't understand how consumer GC trigger kafka server issue: > > java.lang.RuntimeException: A broker is already registered on

Re: Kafka log retentions

A little question， when old message log is deleted， the start point offset is changed. If a simple consumer seek a offset less than start point offset, what will happen? Read all message from start? 2013/4/2 Jason Rosenberg > Essentially, > > There's a configuration property: log.retention.hour

Re: Kafka log retentions

Got it，thanks 2013/4/2 Philip O'Toole > On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao > wrote: > > > A little question， when old message log is deleted， the start point > offset > > is changed. If a simple consumer seek a offset less than start point > >

producer thread safe？

Hi, I want to confirm is private kafka.javaapi.producer.Producer thread safe? i.e. I can use one producer to send data in multi threads at the same time.

InvalidMessageException

Hi, I am using kafka 0.7.2, do you see this exception? What's the possible reason? 2013/05/29 19:18:19.325 ERROR [KafkaRequestHandlers] [] Error processing ProduceRequest on gallery:0 kafka.message.InvalidMessageException: message is invalid, compression codec: NoCompressionCodec size: 222 curr

Re: InvalidMessageException

ggest you to try this. > > Best > > Guodong > > > On Thu, May 30, 2013 at 11:07 AM, Yonghui Zhao >wrote: > > > Hi, > > > > I am using kafka 0.7.2, do you see this exception? What's the possible > > reason? > > > > 2013/05/29 19:1

How to set kafka path in zk

Hi, If I don't want to register kafka in zk root and I want to make it under a namespace, for example kafka1. If I set only one host in zk property something like 10.237.0.1:2181/kafka, it works. But if I set zk property to 3 zk hosts something like 10.237.0.1:2181/kafka,10.237.0.2:2181/kafka,1

How to monitor consumer connection？

Hi, In kafka 0.7.2, we use 2 brokers and 12 consumers. I have 2 questions: 1. seems if one broker is restarted then all consumers need restart to reconnect to kafka. right? 2. How to monitor whether the consumer connection with the broker is healthy? Thank you very much.

Re: Setting the consumer's offset

Hi, How about this feature? thanks *We do plan to allow the high level consumer to specify a starting offset inthe future when we revisit the consumer design. Some of the details aredescribed inhttps://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

offset consume in JAVA api

Hi, I found in 0.9, def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, OffsetAndMetadata], retryOnFailure: Boolean) is added to ZookeeperConsumerConnector. If I want to read data backwards or forwards, can I use this API to commit new offset and then read data from new offset?

message order problem

Broker setting is: 8 partitions, 1 replica, kafka version 0.8.1 We send 2 message at almost same time. Msg1 first, Msg2 second. We have more than 1 producers in sync mode. We may send msg1 in one broker, after *producer.send return response* And send msg2 in the other broker. Both 2 msg has

Re: message order problem

msg2 may not be sent by one broker. 2015-11-21 7:56 GMT+08:00 Guozhang Wang : > Yonghui, > > What is the ack mode for the producer clients? And are msg1 and msg2 sent > by the same producer? > > Guozhang > > On Thu, Nov 19, 2015 at 10:59 PM, Yonghui Zhao > wrote:

Re: offset consume in JAVA api

consumers, hence is lighter-weight. > > > Guozhang > > > On Thu, Nov 19, 2015 at 1:15 AM, Yonghui Zhao > wrote: > > > Hi, > > > > I found in 0.9, > > > > def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, > > OffsetAndMetadat

Re: message order problem

g2 acked. > e. msg1 acked. > > Assuming you said "msg1 and msg2 may notbe sent by one producer", out of > ordering can even happen more likely as they could arrive to the broker > from different sockets at arbitrary ordering. > > Guozhang > > On Mon,

Does high level consumer support setting offset

Hi， In kafka 0.8, does high level consumer support setting offset? Our service reads kafka data but won't flush the data immediately, so if restarted the data in memory will be lost. We want to reset kafka consumer offset to an old offset. If the consumer group has only 1 machine, we can recor

How kafka assign partition to stream

In my environment, I have 2 brokers and only 1 topic, each brokers has 10 partitions, so there are 20 partitions in total. I have 4 consumers in one consumer group, each consumer use createMessageStreams to create 10 streams, 40 streams in total. Since partition can not be split, so there are

zkclient version

Hi I am trying kafka 0.8.1, and use zkclient version: com.101tec zkclient 0.3 But I found exceptions in console, is zkversion is wrong? *java.lang.NoSuchMethodError: org.I0Itec.zkclient.ZkClient.writeDataReturnStat(Ljava/lang/String;Ljava/la

high level consumer shutdown exception

In kafka 0.8.1, when I shut down high level consumer, I found one exception. I think it is expected, right? 14/04/19 21:12:47 INFO consumer.ZookeeperConsumerConnector: [1_yozhao-ubuntu-1397913156352-9b13f962], ZKConsumerConnector shutting down 14/04/19 21:12:47 INFO zookeeper.ClientCnxn: Client s

owner info in zk is not correct

Hi, We are using kafka 0.7. 2 brokers, each broker has 10 partitions for one topic 3 consumers in one consumer group, each consumer create 10 streams. Today, when we want to rollout new service. After we restart one consumer we find exceptions and warning. kafka.common.ConsumerRebalanceFailedE

Re: owner info in zk is not correct

moving the group management like load > rebalance from the ZK-based distributed logic into a centralized > coordiantor. Details can be found here: > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > Guozhang > > > On Mon, May 12,

how to know kafka producer api status

If l use java producer api in sync mode. public void send(kafka.producer.KeyedMessage message) { /* compiled code */ } How to know whether a send process is successful or failed? For example if the kafka broker disk is not accessible , will it throw exceptions?

Re: how to know kafka producer api status

Seems email archive doesn't work now in http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/browser And also I sent email to users-subscr...@kafka.apache.orgto subscribe this group, it doesn't work either. 2014-05-09 16:00 GMT+08:00 Yonghui Zhao : > > If l u