Re: log.retention.size

No, I think more specific settings should get a chance first. I'm suggesting that provided that there is a segment rolled for a topic, *any *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a future log.retention.bytes.global violation would cause segments to be deleted.

Exception in thread "main" java.lang.UnsupportedOperationException: empty.head

Every time when i reboot my machine kafka is not working. when i check the consumeroffset tool i am getting this error. I am new to this i could not figure out what could be the issues. I am saving the logs in /data/zk and /data/kafka. And its running on single node. Would appriciate if any one c

Re: multi-broker topic rebalance

I suspect the issue is that since you have many more partitions than consumers, the data distribution among the partitions is not even. You can take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ?

Re: 答复: how many instances of the broker is suggested in one server.

At LinkedIn, we typically ran one Kafka instance per server. Thanks, Jun On Mon, May 26, 2014 at 9:11 PM, Zhujie (zhujie, Smartcare) < first.zhu...@huawei.com> wrote: > Is there any principles? > > > -邮件原件- > 发件人: Jun Rao [mailto:jun...@gmail.com] > 发送时间: 2014年5月27日 11:25 > 收件人: users@

Re: log.retention.size

For log.retention.bytes.per.topic and log.retention.hours.per.topic, the current interpretation is that those are tight bounds. In other words, only when those thresholds are violated, a segment is deleted. To further satisfy log.retention.bytes.global, the per topic thresholds may no longer be tig

HA: multi-broker topic rebalance

I think you don’t understand my problem... I try to explain in more detail 1. I have a lot of consumers for multiple topics (about 30k)... So one instance of kafka can not handle it 2. I try to connect one part of consumers to one instance of kafka (server 1) and another part to other (

Re: Exception in thread "main" java.lang.UnsupportedOperationException: empty.head

Hi, Could you print the full command line you used consumer offset checker tool? Guozhang On Tue, May 27, 2014 at 12:46 AM, Hemath Kumar wrote: > Every time when i reboot my machine kafka is not working. when i check the > consumeroffset tool i am getting this error. I am new to this i could

Re: java.net.SocketTimeoutException in Broker

Maung, This issue may be due to the session timeout value set too small. With batch size 1000 and message size 420 you are sending 420K of data in each request. What is your time out value? Guozhang On Mon, May 26, 2014 at 11:46 PM, Maung Than wrote: > Hi All, > > This is what we are running

HA: multi-broker topic rebalance

I found also partition reassignment tool but can somebody explain - when I try to execute reassignment what exactly executed? I mean when new partition come to new broker what happened with consumers that connect in old broker 1. they just reconnect to new broker? 2. or they disconnect fr

Re: java.net.SocketTimeoutException in Broker

I think Guozhang meant to say request.timeout.ms, not session timeout. You can try increasing the request timeout through the request-timeout-ms command line option. Thanks, Neha On Tue, May 27, 2014 at 8:55 AM, Guozhang Wang wrote: > Maung, > > This issue may be due to the session timeout va

Re: multi-broker topic rebalance

You can read about the partition assignment tool here. It moves data for a partition's replica from one broker to the other. Thanks, Neha On Tue, May 27, 2014 at 9:04 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan

Re: java.net.SocketTimeoutException in Broker

You meant to say the below parameter; if so we do not override it; so it is the default 10 sec. Should It be larger? It is happening only with Gzip; Snappy or no compression works fine. request.timeout.ms 1 The amount of time the broker will wait trying to meet the request.required.

Re: java.net.SocketTimeoutException in Broker

We are now getting additional IO exception as well: May 27, 2014 12:32:29 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50110] throws java.io.IOException: The server sockets create

Re: reduce replication factor

Yes you can. You can use the partition-reassignment tool and move it to a smaller number of replicas. -Clark Clark Elliott Haskins III LinkedIn DDS Site Reliability Engineer Kafka, Zookeeper, Samza SRE Mobile: 505.385.1484 BlueJeans: https://www.bluejeans.com/chaskins chask...@linkedin.com http

Question on output of kafka-producer-perf-test.sh

Hi All, We are seeing total data sent below is the same with or without compression. Is it always raw data? If so is there a way we can get the compressed volume? start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec [2

Re: java.net.SocketTimeoutException in Broker

What is the new value you set? The new exception you saw seems not related to Kafka network issues, you may want to, for example, check https://netbeans.org/bugzilla/show_bug.cgi?id=14 On Tue, May 27, 2014 at 12:34 PM, Maung Than wrote: > We are now getting additional IO exception as well

Re: Question on output of kafka-producer-perf-test.sh

Maung, Yes, the throughput computed here is based on pre-compression bytes. In the old producer we do not have a metric exposing the compression ratio, in the new producer available in current trunk we can easily add that metric. I have just created a ticket for this a moment ago. https://issues.

Kafka meetup during Hadoop Summit

Hi, We are hosting a Kafka meetup during Hadoop Summit at LinkedIn on June 3rd. Agenda includes the following talks followed by networking. Operating Kafka at Scale by Clark Haskins and Todd Palino@LinkedIn Secure Kafka by Rajasekar Elango@Salesforce Samza - Stream processing at scale on Kafka by

Re: Question on output of kafka-producer-perf-test.sh

Any idea when and which release that could be included in? We would like to have it sooner and can we do something about it? Thanks, Maung On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: > Maung, > > Yes, the throughput computed here is based on pre-compression bytes. In the > old producer

Re: Question on output of kafka-producer-perf-test.sh

It would be probably included in the 0.8.2 release. I have looked into its implementation and it should be in fact a quite simple jira, so we would label it as "newbie" and you are more welcomed to try it on. Guozhang On Tue, May 27, 2014 at 9:16 PM, Maung Than wrote: > Any idea when and which

Re: Exception in thread "main" java.lang.UnsupportedOperationException: empty.head

This is the command i have used. ./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect localhost:2181 --group consumer_group Not just only error with this command. when this sitution occured messages are not able to produce and consumer any more. On Tue, May 27, 2014 at 9:22 PM,

Re: Exception in thread "main" java.lang.UnsupportedOperationException: empty.head

Hemath, I will need more info to check the reasons kafka stopped working. For consumer offset checker tool, you need to specify the topic names, otherwise you will see "empty.head". Guozhang On Tue, May 27, 2014 at 9:53 PM, Hemath Kumar wrote: > This is the command i have used. > > ./kafka-ru