Re: Broker brought down and under replicated partitions

Which version of Kafka are you using? The current stable one is 0.8.1.1 On Tue, Oct 14, 2014 at 5:51 PM, Jean-Pascal Billaud wrote: > Hey Neha, > > so I removed another broker like 30mn ago and since then basically the > Producer is dying with: > > Event queue is full of unsent messages, could n

java api code and javadoc

Hi I downloaded kafka 0.8.1.1 src and went through some documentation and wikis, but could not find any documentation (javadoc or other) on the java API - info on classes like SimpleConsumer, MessageAndOffset etc. Nor could I locate the source code (.java). I see only scala files. Can anyone prov

Re: java api code and javadoc

You probably have to build your own right now. Check out https://github.com/apache/kafka#building-javadocs-and-scaladocs From: 4mayank <4may...@gmail.com> Sent: Wednesday, October 15, 2014 11:38 AM To: users@kafka.apache.org Subject: java api code and java

Re: Broker brought down and under replicated partitions

So I am using 0.8.0. I think I found the issue actually. It turns out that some partitions only had a single replica and the leaders of those partitions would basically "refuse" new writes. As soon as I reassigned replicas to those partitions things kicked off again. Not sure if that's expected...

Re: Achieving Consistency and Durability

Thanks very much for these clarifications, Gwen. I'd recommend modifying the following phrase describing "acks=-1": "This option provides the best durability, we guarantee that no messages will be lost as long as at least one in sync replica remains." The "as long as at least one in sync replica

Missing jars

I added the following dependency in my pom file, however after I add the dependency I get errors: org.apache.kafka kafka_2.10 0.8.1.1 Errors: ArtifactTransferException: Failure to transfer com.sun.jdmk:jmxtools:jar:1.2.1 from Missing artifact com.sun.jmx:jmxri:jar:1.2.1

Re: Consistency and Availability on Node Failures

As I understand it, regardless of whether the partitioning strategy is random or hashed, a producer will eventually try to write a message to one of the partitions that is unavailable (as defined by our acks=-1 and min.isr=2 settings). I suppose that the random partitioner, with enough retries, wi

Live Upgrade

Hi Guys, Can you guys share any experiences you had with Live Upgrade ?. How reliable is it ?.Did we lose messages ?. What issues did you guys face when doing live upgrade. We are planning to upgrade to 0.8.2 from 0.8-beta before we move our web methods broker based messaging layer to Kafka.

[DISCUSS] Release 0.8.2-beta before 0.8.2?

Hi, We have accumulated an impressive list of pretty major features in 0.8.2 - Delete topic Automated leader rebalancing Controlled shutdown Offset management Parallel recovery min.isr and clean leader election In the past, what has worked for major feature releases is a beta release prior to a f

Kafka/Zookeeper deployment Questions

Hello, We're looking into deploying Kafka and Zookeeper into an environment where we want things to be as easy to stand up and administer. To do this, we're looking into using Consul, or similar, and Confd to try to make this as automatic as possible. I was wondering if anyone had an experien

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

If you see the hanging again, it would be great if you can take a thread dump so that we know where it is hanging. Thanks, Jun On Tue, Oct 14, 2014 at 10:35 PM, Abraham Jacob wrote: > Hi Jun, > > Thanks for responding... > > I am using Kafka 2.9.2-0.8.1.1 > > I looked through the controller lo

Re: Cross-Data-Center Mirroring, and Guaranteed Minimum Time Period on Data

One way you can do that is to continually load data from Kafka to Hadoop. During load, you put data into different HDFS directories based on the timestamp. The Hadoop admin can decide when to open up those directories for read based on whether data from all data centers have arrived. Thanks, Jun

Re: Consistency and Availability on Node Failures

Kyle, What you wanted is not supported out of box. You can achieve this using the new java producer. The new java producer allows you to pick an arbitrary partition when sending a message. If you receive NotEnoughReplicasException when sending a message, you can resend it to another partition. Th

0.8.x => 0.8.2 upgrade - live & seamless?

Hi, Some of our SPM users who are eager to monitor their Kafka 0.8.x clusters with SPM are asking us whether the upgrade to 0.8.2 from 0.8.1 will be seamless. I believe this will be the case, but wanted to double-check on that... Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centra

Re: Broker brought down and under replicated partitions

The only thing that I find very weird is the fact that brokers that are dead are still part of the ISR set for hours... and are basically not removed. Note this is not constantly the case, most of the dead brokers are properly removed and it is really just in a few cases. I am not sure why this wou

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

Will definitely take a thread dump! So, far its been running fine. -Jacob On Wed, Oct 15, 2014 at 8:40 PM, Jun Rao wrote: > If you see the hanging again, it would be great if you can take a thread > dump so that we know where it is hanging. > > Thanks, > > Jun > > On Tue, Oct 14, 2014 at 10:35