Inconsistency with Zookeeper

Hi, I am running a Kafka POC with below details * 3 Node cluster (4 Core, 16 GB RAM each) running Kafka 0.8.2.1. * Each node running Kafka & Zookeeper instances. (So total of 3 Kafka brokers & 3 zookeepers) When I tried to create a topic using the kafka-topics.sh, observing the

How to recover from out of sync

Hi guys: we have used kafka-0.8.1 under three machines (broker 0,broker 1,broker 3). After running several months, some partions is out of sync; how to recover from this situation ? topic desc: Topic:analyze PartitionCount:8ReplicationFactor:3 Configs: Topic: anal

Re: FATAL [Replica Manager on Broker XXX]: Error writing to highwatermark file:

Tianning Zhang writes: > > Hi, > > After switching from the Kafka version 2.10-0.8.1.1 to 2.10-0.8.2.1 I frequently encounter the > exception below, which result in re-election of the leaders for the partitions. This exception occurs > every one ~ several hours. > > We have a 3-node-cluster.

Kafka Simple Cosumer

Hi All , Does kafka provides a way to store the offset within zookeper while using the simpleConsumer ? Thanks, Prabhakar

Re: message filterin or "selector"

Is this implemented ? https://cwiki.apache.org/confluence/display/KAFKA/Consumer+API+changes ? This is message filtering on the client or server side ? On Tue, Aug 4, 2015 at 9:54 PM, Gwen Shapira wrote: > The way Kafka is currently implemented is that Kafka is not aware of the > content of mes

Re: Kafka Simple Cosumer

I think you are looking for commitOffsets. I pasted the doc code snippet below: /** * Commit offsets for a topic * Version 0 of the request will commit offsets to Zookeeper and version 1 and above will commit offsets to Kafka. * @param request a [[kafka.api.OffsetCommitRequest]] object.

Re: message filterin or "selector"

The filtering logic there is topic filtering and not message filtering. The idea is to subscribe to multiple topics via a regex whitelist or black list. This does exist today as it does not depend on understanding the content of the message, but I don't think it is what you are looking for. As far

Re: message filterin or "selector"

Is not because of throughput is more about Security. I cant allow all clients to have access to all the topic content (in some cases) I know that access control is something that is not implemented yet, but planed. My idea is to plug a customisation there to add security at selection level too. But

Re: message filterin or "selector"

I think short answer here is that, if you need freeform selectors semantics as per JMS message selectors then you’d need to wrap the API yourself (or get involved in adding the functionality to Kafka). As Gwen and Grant say, you could synthesise something simpler using topics/partitions to pro

Re: message filterin or "selector"

I completely agree with Ben's response. Especially the invitation to propose and get involved in adding functionality to Kafka. A first step to a change this large would be to thoroughly describe your motivations, needed features and proposed changes or architecture in a KIP proposal. This way the

Re: message filterin or "selector"

Yes, thank you very much for the response.. probably building a layer on top of kafka can be the answer. Get involved into add the functionality to kafka could be and option. does anyone knows if there is someone working in something similar so my team can join and help and sponsor the development

Re: message filterin or "selector"

Thanks On Thu, Aug 6, 2015 at 2:20 PM, Grant Henke wrote: > I completely agree with Ben's response. Especially the invitation to > propose and get involved in adding functionality to Kafka. A first step to > a change this large would be to thoroughly describe your motivations, > needed features

Broker side consume-request filtering

Is this discussion open ? Cause this is exactly what I’m looking for.

OffsetOutOfRangeError with Kafka-Spark streaming

Hi, Has anyone tried streaming API of Spark with Kafka? I am experimenting new Spark API to read from Kafka. KafkaUtils.createDirectStream(...) Every now and then, I get following error "spark kafka.common.OffsetOutOfRangeException" and my spark script stops working. I have simple topic with just

Re: Broker side consume-request filtering

Yes - this is what is basically what is termed selectors in JMS or routing-keys in AMQP. My guess is that a KIP to implement this kind of server side filtering would not make it through. Kafka is a producer-centric firehose. Server side filtering wouldn’t really fit well with the original desi

Re: OffsetOutOfRangeError with Kafka-Spark streaming

Does this Spark Jira match up with what you are seeing or sound related? https://issues.apache.org/jira/browse/SPARK-8474 What versions of Spark and Kafka are you using? Can you include more of the spark log? Any errors shown in the Kafka log? Thanks, Grant On Thu, Aug 6, 2015 at 1:17 PM, Cassa

Re: OffsetOutOfRangeError with Kafka-Spark streaming

In Apache Storm some users reported the same issue few months ago [1][2][3]. This was an unusual situation which in our experience only happened when storm topology was asking for offsets that were already trimmed by kafka. Multiple pathological cases(too low retention period, too slow topology, so

Re: OffsetOutOfRangeError with Kafka-Spark streaming

Hi Grant, Yes, I saw exception in Spark and Kafka. In Kafka server logs I get this exception: kafka.common.OffsetOutOfRangeException: Request for offset 2823 but we only have log segments in the range 2824 to 2824. at kafka.log.Log.read(Log.scala:380) at kafka.server.KafkaApis.kafka

Re: OffsetOutOfRangeError with Kafka-Spark streaming

retention.ms is actually millisecond, you want a value much larger then 1440, which translates to 1.4 seconds. On 8/6/15, 4:35 PM, "Cassa L" wrote: >Hi Grant, >Yes, I saw exception in Spark and Kafka. In Kafka server logs I get this >exception: >kafka.common.OffsetOutOfRangeException: Request f

Re: OffsetOutOfRangeError with Kafka-Spark streaming

Looks like this is likely a case very similar to the case Parth mentioned storm users have seen, when processing falls behind the retention period. Perhaps Spark and Kafka can handle this scenario more gracefully. I would be happy to do some investigation/testing and report back with findings and

java.lang.OutOfMemoryError: Java heap space

Hi, I am trying to integrate https://github.com/relango/kafka/tree/kafka_security_0.8.2 with Spark Streaming using the SimpleConsumer. I know that the SSL patch is on its way but need to set up a prototype hence went ahead with Raja's version. So when I run my spark job to retrieve data from 1 to

Re: java.lang.OutOfMemoryError: Java heap space

Hi Sourabh, We have seen this error, if kafka broker was running with SSL on Consumer is trying to consumer in plaintext mode. Are you using high level consumer or SimpleConsumer..? If you using using SimpleConsumer, pull latest code from my repo

Re: java.lang.OutOfMemoryError: Java heap space

I am using SimpleConsumer and checked out the code from your repo 1.5 weeks back, did you push anything post that? Thanks, Sourabh On Thursday, August 6, 2015, Rajasekar Elango wrote: > Hi Sourabh, > > We have seen this error, if kafka broker was running with SSL on Consumer > is trying to cons

Re: How to recover from out of sync

Hi, guys: I have fixed the problem just restart kafka broker 0… According to http://qnalist.com/questions/4968010/isr-not-updating >>> Auto leader balancing has some know issues when using together with >>> controll

kafka log flush questions

Hi , I am trying to understand the Kafka log flush behavior. My understanding is when the broker specifies broker config param "log.flush.interval.ms", it will specify log config param "flush.ms" internally. In logManager logic, when the log exceed flush.ms, it will call Log.flush which will call