Re: 0.9.0 release branch

Thanks, Jun. We will switch clients shortly after the release. Cliff > On Nov 2, 2015, at 7:26 PM, Jun Rao wrote: > > Cliff, > > We try not to patch the old consumer too much since we are adding the new > java consumer in 0.9. The new consumer supports callbacks during rebalances > and can ad

Re: Problems getting offsets

Hi David, My Bad. I did not understand your question correctly. Thanks Stevo for detailed explanation. Just incase, if you want checkout kafka-based offset management, this is a very good presentation : http://www.slideshare.net/jjkoshy/offset-management-in-kafka Thanks, Mayuresh On Mon, Nov 2

Re: 0.9.0 release branch

Cliff, We try not to patch the old consumer too much since we are adding the new java consumer in 0.9. The new consumer supports callbacks during rebalances and can address the problem in KAFKA-2725 better. Thanks, Jun On Mon, Nov 2, 2015 at 11:16 AM, Cliff Rhyne wrote: > Hi Jun, > > I openne

Re: 0.9.0 release branch

I added KAFKA-2698 as an 0.9.0.0 blocker. It adds an API to query the currently paused partitions. Here's the PR: https://github.com/apache/kafka/pull/403 On Mon, Nov 2, 2015 at 11:16 AM, Cliff Rhyne wrote: > Hi Jun, > > I openned KAFKA-2725 bas

Re: Controller sometimes lose ISR

The broker 61 some how falls behind in fetching from the leader brokers and hence falls out of the ISR. [2015-10-30 23:02:34,012] ERROR Controller 61 epoch 2233 initiated state change of replica 61 for partition [test-res-met.server_logs.conv,18] from OnlineReplica to OfflineReplica... means that

Re: 0.9.0 release branch

Hi Jun, I openned KAFKA-2725 based on my experience with duplicate message processing with auto-commit off. I think it's a fairly small change, especially for someone familiar with the kafka code-base but it makes a big impact for clients not using auto-commit. Can this be included in 0.9.0? Th

Re: Migrating both Zookeeper ensemble and Kafka cluster please confirm steps

Hi Tulio, Thanks a bunch for responding to my question! We basically need to migrate data from brokers hosted on racks being decommisioned, but the new and old brokers are actually within the same cluster. The new brokers are configured to use the new Zookeeper ensemble, but, again, they are inten

Re: Migrating both Zookeeper ensemble and Kafka cluster please confirm steps

I'm not really sure but I think kafka-reassing-partitions.sh only works to move partitions between instanses of the same cluster If you have two separete clusters, it is not going to work. Are you using two different zookeeper clusters, or simply adding instance to the existing one? On Sun, Nov

Re: 0.9.0 release branch

I added KAFKA-2691 as well, which improves client handling of authorization errors. -Jason On Mon, Nov 2, 2015 at 10:25 AM, Becket Qin wrote: > Hi Jun, > > I added KAFKA-2722 as a blocker for 0.9. It fixes the ISR propagation > scalability issue we saw. > > Thanks, > > Jiangjie (Becket) Qin > >

Re: High delay during controlled shutdown and acks=-1

Hi Federico, What is your replica.lag.time.max.ms? When acks=-1, the ProducerResponse won't return until all the broker in ISR get the message. During controlled shutdown, the shutting down broker is doing a lot of leader migration and could slow down on fetching data. The broker won't be kicked

Re: 0.9.0 release branch

Hi Jun, I added KAFKA-2722 as a blocker for 0.9. It fixes the ISR propagation scalability issue we saw. Thanks, Jiangjie (Becket) Qin On Mon, Nov 2, 2015 at 9:16 AM, Jun Rao wrote: > Hi, everyone, > > We are getting close to the 0.9.0 release. The current plan is to have the > following remai

Re: Quick kafka-reassign-partitions.sh script question

Nice! Thanks Gwen! --John On Mon, Nov 2, 2015 at 1:03 PM, Gwen Shapira wrote: > Actually, no. You can move partitions online. > > The way it works is that: > 1. A new replica is created for the partition in the new broker > 2. It starts replicating from the leader until it catches up - if you >

Re: Quick kafka-reassign-partitions.sh script question

Actually, no. You can move partitions online. The way it works is that: 1. A new replica is created for the partition in the new broker 2. It starts replicating from the leader until it catches up - if you continue producing at this time, it will take longer to catch up. 3. Once the new replica ca

Re: Kafka and Spark Issue

Since the error is from the HBase client and completely unrelated to Kafka, you will have better luck in the HBase mailing list. On Mon, Nov 2, 2015 at 9:16 AM, Nikhil Gs wrote: > Hello Team, > > My scenario is to load the data from producer topic to Hbase by using Spark > API. Our cluster is Ke

Kafka and Spark Issue

Hello Team, My scenario is to load the data from producer topic to Hbase by using Spark API. Our cluster is Kerberos authenticated and when we running the below kafkaToHbase.java and the error which I am facing is also below. Let me know if anyone can have any idea what can be done. package com.s

0.9.0 release branch

Hi, everyone, We are getting close to the 0.9.0 release. The current plan is to have the following remaining 0.9.0 blocker issues resolved this week, cut the 0.9.0 release branch by Nov. 6 (Friday) and start the RC on Nov. 9 (Monday). https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAF

Re: Topic per entity

Hi Alex & Andrew, There was a discussion with some pointers on this mailing list a bit ago titled "mapping events to topics". I suggest taking a look at that thread: http://search-hadoop.com/m/uyzND1vJsUuYtGD91/mapping+events+to+topics&subj=mapping+events+to+topics If you still have questions, do

Error Producer

Hi guys, How could I solve this problem? %Failed to produce message: Local: Queue full Thanks

High delay during controlled shutdown and acks=-1

Hi, I have few java async producers sending data to a 4-node Kafka cluster version 0.8.2, containing few thousand topics, all with replication factor 2. When i use acks=1 and trigger a controlled shutdown + restart on one broker, the producers will send data to the new leader, reporting a very low

Quick kafka-reassign-partitions.sh script question

Hi Everyone, Perhaps a silly question...does one need to shut down incoming data feeds to Kafka prior to moving partititions via the kafka-reassign-partitions.sh script? My thought is yes, but just want to be sure. Thanks --John

Re: Problems getting offsets

Here is a bit longer and more detailed, not necessarily better understandable explanation. When using Kafka for offsets storage, consumer offsets get stored in (compacted) consumer offsets Kafka topic ("__consumer_offsets"). Every partition of consumer offsets topic could store offsets from multip