Re: Max Message Size

There is a configurable max message size on the broker. Oversized messages will be rejected. Thanks, Jun On Tue, May 13, 2014 at 11:23 AM, Bhavesh Mistry wrote: > Hi Kafka Team, > > Is there any message size limitation from producer side ? If there, is > what happens to message, does it get

Kafka 0.7.X maven repo

We are currently using kafka 0.8.X but want to try something with 0.7.x and compare. Can anyone point me to the kafka 0.7.X maven repo ? Thanks in advance Vijay

Re: Controller to broker connection timing out through load balancer

Which version of Kafka are you using? Thanks, Jun On Tue, May 13, 2014 at 5:56 PM, Robin Yamaguchi wrote: > It seems like this mailing list wasn't updating through the web archives > for a few days last week, so I wanted to send this out again in case it > wasn't seen. My apologies for the re

kafka performance question

our version is kafka_2.10-0.8.1 发件人: Zhujie (zhujie, Smartcare) 发送时间: 2014年5月14日 8:56 收件人: 'users@kafka.apache.org'; 'd...@kafka.apache.org' 主题: kafka performance question Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower than we want. In our

Re: Killing last replica for partition doesn't change ISR/Leadership if replica is running controller

Yes, that seems like a real issue. Could you file a jira? Thanks, Jun On Tue, May 13, 2014 at 11:58 AM, Alex Demidko wrote: > Hi, > > Kafka version is 0.8.1.1. We have three machines: A,B,C. Let’s say there > is a topic with replication 2 and one of it’s partitions - partition 1 is > placed on

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

Hi Yury, When auto-commit is turned on, before the consumer shuts down completely it needs to commit its last offset to ZK to avoid any duplicates consumed when it starts over. If the ZK is not up running then the shutdown would rather stuck than let it go and incur duplicates. This behavior is by

Re: keyed-messages & de-duplication

Hi, The compaction is done to clean-up space. It isn't done immediately only periodically. I suspect the reason you see no compaction is that we never compact the active segment of the log (the most recent file) as that is still being written to. The compaction would not happen until a new segmen

Re: Loss of Leader in Kafka

After topic deletion, I deleted the physical folders in data logs and then restarted Kafka. That resolved. Thanks! Will try out 0.8.1 and let the group know. Regards, Kashyap On Monday, May 12, 2014, Jun Rao wrote: > Delete topic doesn't quite work in 0.8.1. We recently fixed it in trunk. > Cou

kafka performance question

Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower than we want. In our cluster,there is a provider and a broker. We use a one thread read file from local disk of provider and send it to broker. The average throughput is only 3 MB/S~4MB/S. But

Re: 100 MB messages

It can, but it will not perform very well. Kafka fully instantiates messages in memory (as a byte[] basically) so if you send a 100MB message the server will do a 100MB allocation to hold that data prior to writing to disk. I think MongoDB does have blob support so passing a pointer via Kafka as y

0.7 maven repo

We are currently using kafka 0.8.X but want to try something with 0.7.x and compare. Can anyone point me to the kafka 0.7.X maven repo ? Thanks in advance Vijay

zookeeper down

Can kafka survive when zookeeper is down and not connectable ? Will the consumer or producer still work in that case ? Weide

Re: Kafka 0.7.X maven repo

Maven wasn't introduced until 0.8.0. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop /

Re: Loss of Leader in Kafka

Hi Kashyap, This may be a real issue with delete-topic feature, could you also reproduce this with trunk HEAD and 1 partition/1 replica? Guozhang On Mon, May 12, 2014 at 9:39 AM, Kashyap Mhaisekar wrote: > Hi, > I am hitting a strange exception while creating a topic in Kafka - > Steps to gene

Killing last replica for partition doesn't change ISR/Leadership if replica is running controller

Hi, Kafka version is 0.8.1.1. We have three machines: A,B,C. Let’s say there is a topic with replication 2 and one of it’s partitions - partition 1 is placed on brokers A and B. If the broker A is already down than for the partition 1 we have: Leader: B, ISR: [B]. If the current controller is n

Re: 100 MB messages

Or HDFS and use Kafka for the event of file, yup. Processing on the files can be done without the mapreduce overhead in Hadoop now using Apache Tez (or something that use Tez like Pig). /*** Joe Stein Founder, Principal Consultant Big Data Open Sourc

Re: TCP_TOO_MANY_ESTAB_CONNS

Hi Ranjith, Which version of Kafka are you using? Guozhang On Tue, May 13, 2014 at 3:53 AM, Ranjith Venkatesan wrote: > Dear all, > > We started using kafka as a queue server recently. Our production > setup has 3 kafka servers and 3 zookeepers. We having 30 topics with 10 > partitions

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

In 0.8, ZK is critical for managing the brokers. So, we do expect that ZK service to be always available. Thanks, Jun On Mon, May 12, 2014 at 9:59 AM, Yury Ruchin wrote: > Hi all, > > I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. > Steps to reproduce: > > 1. Start sin

100 MB messages

Hello everyone, Can Kafka be used for binary large objects of 100 MB ? Or should I use a different solution to store these files like MongoDB and maybe send the location of these files in MongoDB over Kafka? Thanks is advance, Wouter

What happens to Kafka when ZK lost its quorum or becomes unstable?

Hi, Can the producers, brokers and consumers still be processing messages when their ZK cluster lost its quorum or becomes unstable? I know this is rather general question as it may depends on what configuration these use. So, please enumerate all of those combinations. Thanks, Connie

Re: TCP_TOO_MANY_ESTAB_CONNS

Which version of Kafka are you using? On Tue, May 13, 2014 at 3:53 AM, Ranjith Venkatesan wrote: > Dear all, > > We started using kafka as a queue server recently. Our production > setup has 3 kafka servers and 3 zookeepers. We having 30 topics with 10 > partitions each. We have producer

Re: Loss of Leader in Kafka

>> 3. Deleted the topic. Checked only the zookeeper to see if the /brokers/topics DOES NOT have the topic You are seeing this problem since delete topic is not supported in Kafka. Any attempt to delete a topic may leave your cluster in an unstable state. We plan to release 0.8.2 with delete topic

Re: What happens to Kafka when ZK lost its quorum or becomes unstable?

The brokers are designed with the expectation that ZK service is always available. If ZK is not available, the clients may not be able to send/fetch data properly. Thanks, Jun On Tue, May 13, 2014 at 6:48 AM, Connie Yang wrote: > Hi, > > Can the producers, brokers and consumers still be proce

Re: Kafka producer in CSharp

The wire protocol has changed drastically since then. I don't know of any C# clients (there are none on the client library page nor have I heard of any being used in production but maybe there are some). For clients that use DotNet I often suggest that they use some HTTP producer/consumer https:

What happens to Kafka when ZK lost its quorum?

Hi all, Can Kafka producers, brokers and consumers still be processing messages and functioning in their normal states if Zookeeper lost its quorum? Thanks, Connie

TCP_TOO_MANY_ESTAB_CONNS

Dear all, We started using kafka as a queue server recently. Our production setup has 3 kafka servers and 3 zookeepers. We having 30 topics with 10 partitions each. We have producer pool(10 producers / APP_SERVERS). So totally 30 producers. With our monitoring tool, we could detect "TCP

Re: Kafka producer in CSharp

Ok got some info myself. I can have fault tolerance - I can start kafka-http-endpoint using broker lists I can have ack - start using --sync But what is best practice in case if kafka-http-endpoint goes down? Start multiple kafka-http-endpoint's and in client side just control that kafka-htt

Re: Kafka producer in CSharp

Thank you for response. I think HTTP is ok. I have two more question in case of HTTP. 1. Can I have fault tolerance in case I have two or more brokers? 2. Can I ack that message is in queue? Best regards, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee http://ee.linkedin.com/in/margusro