How to build 0.8.1 from source

Good morning. I am trying to build 0.8.1 from source because I like to include some jars that are not part of official distribution. I cloned 0.8.1 tag git clone -b 0.8.1.0 https://git-wip-us.apache.org/repos/asf/kafka.git. The issue I have its missing sbt. How can I build from source? Please advis

Re: How to build 0.8.1 from source

We moved to gradle in 0.8.1. See the README for instructions on gradle build - https://github.com/apache/kafka/blob/0.8.1/README.md Thanks, Neha On Mon, Mar 24, 2014 at 10:07 AM, Vadim Keylis wrote: > Good morning. I am trying to build 0.8.1 from source because I like to > include some jars tha

Re: How to build 0.8.1 from source

Thanks so much Neha. On Mon, Mar 24, 2014 at 10:13 AM, Neha Narkhede wrote: > We moved to gradle in 0.8.1. See the README for instructions on gradle > build - https://github.com/apache/kafka/blob/0.8.1/README.md > > Thanks, > Neha > > > On Mon, Mar 24, 2014 at 10:07 AM, Vadim Keylis >wrote: > >

Memory consumption in Kafka

Hi, We have been doing some evaluation testing against Kafka.We have 48GB RAM on each broker. I created 3 broker cluster and one zookeeper and sent 10,000Messages/seconds to this cluster, continuously. Payload is very small, less than Kilobyte. Consumer was readind 5,000Messages per second. Wha

Re: Memory consumption in Kafka

How are you measuring memory usage? I would expect the OS page cache to take 100% of unused memory, but that's not the same as being OOM. - Niek On Mon, Mar 24, 2014 at 12:38 PM, Cassa L wrote: > Hi, > We have been doing some evaluation testing against Kafka.We have 48GB RAM > on each broker.

Re: Kafka High Level Consumer Connector shuts down after 10 seconds

Hi Gufran, Does the consumer has any messages to consume during these 10 seconds? There is also another config called "socket.timeout.ms", and if that amount of time has elapsed when there is no data coming back through the socket, the consumer will close automatically. From your logs it seems the

questions about mirror maker

Hi, 1. In documentation 0.8.1, the mirror maker command is: bin/kafka-run-class.sh kafka.tools.MirrorMaker --zookeeper zk_host:port/chroot --consumer.config consumer-1.properties --consumer.config consumer-2.properties --producer.config producer.properties --whitelist my-topic How

Re: questions about mirror maker

Hi Churu, The documentations needs to be updated. Basically, you need to specify the producer and consumer config files as part of the parameters: bin/kafka-run-class.sh kafka.tools.MirrorMaker Missing required argument "[consumer.config]" Option Description -- --- --blacklist

Reinstating ephemeral nodes and watchers on zk session timeout

Hi On zookeeper session timeout due to some stopping the world long GC pause or zookeeper server outage, Ephemeral nodes on kafka broker and consumer should be recreated but in my test environment, handleNewSession() is not called. My test scenario is, starting kafka broker locally and put a brea

Help in setting up Kafka cluster

Hi, First time I am trying to setup new kafka cluster. I have tried sudo cluster with cli based kafka producer and consumer. Having difficulties with setting up log aggregation kafka producer and consumer. I would appreciate if anyone can help me in this. - roy

Log size and retention

Hi All, I'm trying to tune the log retention size and have a question. I have a replication factor of 3 on a cluster of 5 brokers. How many partitions will a broker host? Is there any way to tell? Or do I have to assume that each broker will host all partitions and size accordingly? Thanks much

Re: New Consumer API discussion

Hey Chris, Really sorry for the late reply, wonder how this fell through the cracks. Anyhow, thanks for the great feedback! Here are my comments - 1. Why is the config String->Object instead of String->String? This is probably more of a feedback about the new config management that we adopted in

Re: New Consumer API discussion

I took some time to write some example code using the new consumer APIs to cover a range of use cases. This exercise was very useful (thanks for the suggestion, Jay!) since I found several improvements to the APIs to make them more usable. Here are some of the changes

Re: Reinstating ephemeral nodes and watchers on zk session timeout

I think you are trying to introduce a session expiration, then could you try to do the following and see if you can reproduce the session expiration? ./bin/kafka-server-start.sh config/server.properties kill -SIGSTOP sleep 6s At this point, the session will be expired and the node will disappear

Re: Help in setting up Kafka cluster

Could you please provide more details on what problems you run into while setting up Kafka to do log aggregation? On Mon, Mar 24, 2014 at 3:35 PM, , Roy wrote: > Hi, > > First time I am trying to setup new kafka cluster. I have tried sudo > cluster with cli based kafka producer and consumer.

Fwd: When running kafka-preferred kafka-preferred-replica-election.sh for a second time a node exists exception is being thrown

When running kafka-preferred kafka-preferred-replica-election.sh for a second time a node exists exception is being thrown for /admin/preferred_replica_election The node doesn't seem to get cleaned up. Failed to start preferred replica election kafka.common.AdminCommandFailedException: Admin comma

RE: Fwd: When running kafka-preferred kafka-preferred-replica-election.sh for a second time a node exists exception is being thrown

Yes, I also have this problem using 0.8.1. if you want second time is works, I always manual delete the node in zookeeper Best Regards Jerry -Original Message- From: "Chris George" To: ; Cc: Sent: 2014-03-25 (星期二) 05:03:22 Subject: Fwd: W

Re: Memory consumption in Kafka

Are the following jmx values high on the brokers? "kafka.server":name="PurgatorySize",type="ProducerRequestPurgatory" "kafka.server":name="PurgatorySize",type="FetchRequestPurgatory" Thanks, Jun On Mon, Mar 24, 2014 at 12:38 PM, Cassa L wrote: > Hi, > We have been doing some evaluation testi

Re: Kafka High Level Consumer Connector shuts down after 10 seconds

Java threadpool is notorious for eating exceptions. Could you add try/catch in ConsumerTest while iterating through the messages? Thanks, Jun On Mon, Mar 24, 2014 at 12:44 PM, Gufran Pathan wrote: > Hi, > > > > I'm facing an issue exactly similar to the one issued by someone else a > few days

Re: questions about mirror maker

Updated the doc in 0.8.1. Thanks, Jun On Mon, Mar 24, 2014 at 2:44 PM, Guozhang Wang wrote: > Hi Churu, > > The documentations needs to be updated. > > Basically, you need to specify the producer and consumer config files as > part of the parameters: > > bin/kafka-run-class.sh kafka.tools.Mir

Re: When running kafka-preferred kafka-preferred-replica-election.sh for a second time a node exists exception is being thrown

Yes, the output is unintuitive. The rebalancing ran asynchronously. You can't trigger another one until the previous one completes. Could you file a jira for this? Thanks, Jun On Mon, Mar 24, 2014 at 1:03 PM, Chris George wrote: > When running kafka-preferred kafka-preferred-replica-election.s

Re: Log size and retention

Does the following help? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic ? Thanks, Jun On Mon, Mar 24, 2014 at 4:26 PM, Tom Amon wrote: > Hi All, > > I'm trying to tune the log retention size and have a question. > > I have a replication f

Re: Memory consumption in Kafka

Hi, I did not check above values for "PurgatorySize". Here is the output of free -m when test was running. total used free sharedbuffers cached Mem: 48356 48217139 0419 44584 -/+ buffers/cache: 3212 45144

Question about manual tracking of Offset

Hi experts & Kafka Dev team, Have a very quick question and need your help in designing a consumer. I am trying to keep the email short and simple. Scenario, Lets say: 1) Have a Kafka with 20 messages(from Offset 0 - 19) 2) Pulling the 1st 10 message from 0th to 9th Offset(assuming the desired

Question about manual tracking of Offset

Hi experts & Kafka Dev team, Have a very quick question and need your help in designing a consumer. I am trying to keep the email short and simple. Scenario, Lets say: 1) Have a Kafka with 20 messages(from Offset 0 - 19) 2) Pulling the 1st 10 message from 0th to 9th Offset(assuming the desired