Controller is not being failed over 0.8.1

Hi, We got the following error spamming the logs when broker 1 is the controller and we are shutting it down in controlled manner not kill -9. The leader being switched to broker 2 for all partitions but controller is not being failed over to broker 2. [2014-04-16 10:48:47.976-0600] ERROR [Con

Cluster design distribution and JBOD vs RAID

I am wondering what others are doing in terms of cluster separation. (if at all) For example let’s say I need 24 nodes to support a given workload. What are the tradeoffs between a single 24 node cluster vs 2 x 12 node clusters for example. The application I support can support separation of data

enhancements to kafka-websocket

i've been a busy little bee this week so kafka-websocket now supports keyed messages and user-defined transforms for messages. the latter is pretty raw, improvement suggestions in the form of pull requests welcome! https://github.com/b/kafka-websocket/commit/6430fd522c0e0305995f648511726da0b616d2e

jruby-kafka and logstash-kafka nearly feature complete

Hi everyone, I have updated my jruby-kafka library to support pretty much all the default consumer and producer classes to support all the options included in the docs. Check it out, give some feedback: https://github.com/joekiller/jruby-kafka This library is primarily supporting my logstash

How to calculate number of partitions?

Hi, I am sizing Kafka setup for the first time. One question I have is, how to determine number of partitions my topic will need. Are there any considerations to be taken in order to determine this number. I am planning to have a three node Kafka cluster. LCassa.

leaders being kicked out of ISR

Hey all, This is xinyao from dropbox and I am trying to push kafka to widely used inside dropbox. I setup up five nodes with 2x replication. The system is running well to handle all the traffic. The problem I am seeing now is that one host(probably one leader) being kicked out of ISR. This is som

Re: How to calculate number of partitions?

Does this help: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic? On Wed, Apr 16, 2014 at 03:34:28PM -0700, Cassa L wrote: > Hi, > I am sizing Kafka setup for the first time. One question I have is, how > to determine number of partitions my

RE: Cluster design distribution and JBOD vs RAID

Perhaps as you consider the size of your cluster, a few questions about the kind of messaging you are looking at? I can use an example of what we do in our production environment while not going into specifics. These are just observations from an OPS perspective. (sorry for the wall of text.) *

Re: leaders being kicked out of ISR

Adding some context. The lost leader is keep trying to write: [2014-04-16 16:53:10,126] ERROR Conditional update of path /brokers/topics/photos-mobile_analytics_user_logs-0060/partitions/0/state with data {"controller_epoch":1,"leader":1,"version":1,"leader_epoch":0,"isr":[1]} and expected versio

Re: Controller is not being failed over 0.8.1

What's the controller value in the zk path (see https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)? Any error in the controller/state-change log? Thanks, Jun On Wed, Apr 16, 2014 at 10:07 AM, Seshadri, Balaji wrote: > Hi, > > We got the following error spamm

Re: jruby-kafka and logstash-kafka nearly feature complete

Joseph, Thanks for the update. Jun On Wed, Apr 16, 2014 at 2:26 PM, Joseph Lawson wrote: > Hi everyone, > > > I have updated my jruby-kafka library to support pretty much all the > default consumer and producer classes to support all the options included > in the docs. Check it out, give som

Re: leaders being kicked out of ISR

Any error in the controller/state-change logs? Do you have GC induced ZK session expiration in the brokers? Thanks, Jun On Wed, Apr 16, 2014 at 9:50 AM, Xinyao Hu wrote: > Hey all, > > This is xinyao from dropbox and I am trying to push kafka to widely used > inside dropbox. > > I setup up fi