Re: How to define multiple serializers in kafka?

I think by design it would be better to put different kind of messages in a different topic. But if you would want to mix you can make your own serializer/deserializer you could append a 'magic byte' to the byes you get after you serialize, to be able to deserialize using the correct methods. The c

Re: Kafka as Transaction-log - Producer fencing

The key feature of BookKeeper in this case is that it guarantees that exactly only one "leader" can change the state of the machine, you can think about it as a way of implementing a consensus algorithm. For instance if the "old" leader (Kafka producer in this case) succeeds to write to the log

Re: Filter plugins in Kafka

If you want something really simple and hacky, you could use kafkatee[1] and kafkacat[2] together: kafkatee.conf: input [encoding=string] pipe tail -f a.log output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic [1] https://github.com/wikimedia/analytics-kafkatee [2] https://github

Use OCSP in Kafka

Hi, I'd like to know if its doable to config kafka ssl to that I can use OCSP to revoke certificates. My use case is a kafka broker(s) to which consumers will connect using ssl (authentication + authorization). I will provide the certificates to my consumers but I want to be sure I can revoke cer

Topic retention configuration clarification

Using 0.9.0.1. I'm building a new topic that should keep data for much longer than the brokers' default, say at least a year, before deleting messages. http://kafka.apache.org/documentation.html says setting the "retention.ms" for the topic will adjust the time, but I cannot find out what uni

Re: Topic retention configuration clarification

Hi Lawrence, Why do you need so much retention? We've generally found that all use of Kafka that wants really long retention (e.g. for compliance or replay reasons) are better served by consuming from the topic and putting the data on S3 (or some other longer term storage) for anything beyond a fe

kafka producer 0.9.0 - performance degradation

We are writing messages at the rate of about 9000 records/sec into our kafka cluster, at times we see that the producer performance degrades considerably and then it never recovers. When this happens we see the following error "unable to allocate buffer within timeout". The "waiting-threads" metric

Archive feature

Hi all, There was some discussion on the list a while back about a possible feature to archive log segments when they expire rather than deleting them. Did anything like that ever become a realization? Thanks -- *This e-mail is not encrypted. Due to the unsecured nature of unencrypted e-mail,

Re: Archive feature

Hi Scott, I remember that one. That would be an awesome feature. If there was anybody wanting to help with contributing such thing, I’ll be happy to dig the details out. Best regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com)

Re: leader election bug

We are running Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT, does it have any known problems? On Fri, Apr 29, 2016 at 2:35 PM, James Brown wrote: > What version of ZooKeeper are you on? There have been a few bugs over > the years where ZK has lost ephemeral nodes (and spontane

Re: leader election bug

Also that broker is not registered in ZK as we can check with zk-shell, but kafka still thinks it's a leader for some partitions. On Mon, May 2, 2016 at 11:04 AM, Kane Kim wrote: > We are running Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 > GMT, does it have any known problems?

Saprk Consumer skips offset

Hi, We are running a Kafka consumer using Spark consumer based approach with redis to store offset. We have set a message read rate as 200 per sec with 5min as interval. We are facing some issues with offset while streaming the data. The consumer skip offsets in some iterations, mostly when its no

RE: Invalid TimeStamp Error while running WordCountDemo - kafka-0.10.0

Guozhang, That worked!! Thanks a lot for your timely response. See you with another issue. Regards, Buvaan -Original Message- From: EXT Guozhang Wang [mailto:wangg...@gmail.com] Sent: Sunday, May 01, 2016 8:15 PM To: users@kafka.apache.org Subject: Re: Invalid TimeStamp Error while runn

Re: Encryption at Rest

Hello, We tried encrypting the data before sending it to kafka, however this makes the compression done by kafka almost impossible. Also the performance overhead of encrypting the individual messages was quite significant. Ideally, a pluggable “compression” algorithm would be best. Where messag

Re: Encryption at Rest

I'm trying to understand your use-case for encrypted data. Does it need to be encrypted only over the wire? This can be accomplished using TLS encryption (v0.9.0.0+). See https://issues.apache.org/jira/browse/KAFKA-1690 Does it need to be encrypted only when at rest? This can be accomplished usin

Re: leader election bug

Thats a good version :) On Mon, May 2, 2016 at 11:04 AM, Kane Kim wrote: > We are running Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 > GMT, does it have any known problems? > > On Fri, Apr 29, 2016 at 2:35 PM, James Brown wrote: > >> What version of ZooKeeper are you on? There h

Re: leader election bug

So what could happen then? There is no broker registered in zookeeper, but it's still a leader somehow. On Mon, May 2, 2016 at 3:27 PM, Gwen Shapira wrote: > Thats a good version :) > > On Mon, May 2, 2016 at 11:04 AM, Kane Kim wrote: > > We are running Zookeeper version: 3.4.6-1569965, built o

Why the consumer not listening to the first message?

Hi all; In my test program,I start listener. Then sending messages in a loop. If i send one message, it is not listning that message. If i send 2 messages, it listens one message.If I send 3 , it listens 2 messages..Why is that? *Producer* *KeyedMessage message = new KeyedMessage(topic, serialize

Re: How to define multiple serializers in kafka?

What is the best way for this? Do we need to have common serializer/deserializer for all type of the objects we publish? OR seperate for each objects? If we have seperate serializer/deserializers, then how can I configure kafka? Or Is it recommended to use Avro schemas? Thanks On 2 May 2016 at 18

Re: Why the consumer not listening to the first message?

I fixed this by setting following property in my producer. request.required.acks=1 On 3 May 2016 at 09:50, Ratha v wrote: > Hi all; > In my test program,I start listener. Then sending messages in a loop. > If i send one message, it is not listning that message. If i send 2 > messages, it liste

Receiving ILLEGAL_GENERATION, but I can't find information on the exception.

I'm receiving the following exception, but I'm unable to make heads or tails of the exception or find informaton on it. "Error ILLEGAL_GENERATION occurred while committing offsets for group graph-consumer" The Java class doesnt contain information on it either. e.g., IllegalGenerationException I'

Re: Receiving ILLEGAL_GENERATION, but I can't find information on the exception.

It means there was a consumer group rebalance that this consumer missed. You may be spending too much time in msg processing between poll() calls. -Dana

Re: Receiving ILLEGAL_GENERATION, but I can't find information on the exception.

To add to what Dana said, we fixed this issue on AWS with setting the “max.partition.fetch.bytes” to a smaller setting so out consumer would poll more frequently. Try setting “max.partition.fetch.bytes” to “75”, then “50”, then “25”, … until the error stop occurring. The default i

Getting Timed out reading socket error for kafka cluster setup

Hi, I want to setup a kafka cluster type setup for three similar application having same queues like AppA -> {TopicX, TopicY, TopicZ}, AppB -> {TopicX, TopicZ}, AppC -> {TopicX, TopicY}. Producer and Consumer will be App specific. I setup kafka cluster with three brokers having partition 1,2,3 in

Re: Encryption at Rest

We did try indeed the last scenario you describe as encrypted disks do not fulfil our requirements. We need to be capable of changing encryption keys on regular intervals and in case of expected key compromise. Also, when a running machine is hacked, disk based or file system based encryption do

Re: Invalid TimeStamp Error while running WordCountDemo - kafka-0.10.0

Hi Buvana, Actually Ismael just pointed out that the new KafkaProducer already set the timestamp as current wall-clock time. And I cannot reproduce this issue from trunk either. Did you ran the console producer that is built from the trunk April 22nd as well? Guozhang On Mon, May 2, 2016 at 12:3

MirrorMaker consumers getting stuck on certain partitions

Hi, I am running Mirror Maker (version 0.9 , new consumer). Sometimes I find that consumer gets stuck on certain partitions and offset doesn’t move in that case. I have 10 MM process running, each having 10 streams. The topic has 100 partitions. Below is the sample output of the consumer (I ha

Re: Encryption at Rest

"We need to be capable of changing encryption keys on regular intervals and in case of expected key compromise." is achievable with full disk encryption particularly if you are willing to add and remove Kafka servers so that you replicate the data to new machines/disks with new keys and take the ma

FW: MirrorMaker consumers getting stuck on certain partitions

Consumption on Partitions – 99,0 and 98 are getting stucked. Thanks Tushar From: "Mhaskar, Tushar" mailto:tmhas...@paypal.com>> Date: Monday, May 2, 2016 at 9:52 PM To: "users@kafka.apache.org" mailto:users@kafka.apache.org>> Subject: MirrorMaker consumers getting

Re: Encryption at Rest

From what I understand, when using batch compression in Kafka, the files are stored compressed. Don’t really see the difference between compression and encryption in that aspect. If Kafka would support pluggable algorithms for compression (it already supports two), it would be rather straightfor