Re: Messages are repeating in kafka

@Kant Actually, after setting that configuration I deleted a topic having 100 partitions. After that, I checked through ./kafka-topic.sh --list command. It was not showing that topic. Again I started producers and ran the following command *./kafka-consumer-groups.sh --bootstrap-server localhost:9

Re: Question regarding buffer.memory, max.request.size and send.buffer.bytes

As the document suggest, you should keep it same as your max.request.size. And the documentation clearly says that the system will throw exception if you are producing faster than you can send, and kept blocking until max.block.ms is reached. On 24 May 2017 at 00:51, Milind Vaidya wrote: > I a

Re: Streams error handling

Just a heads up that we're tracking this and other improvements in exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 . Thanks Eno > On 23 May 2017, at 17:31, Mike Gould wrote: > > That's great for the value but not the key

Kafka Streams and AWS IOPS Credits

Hi All, In the past, we experienced lots of problems with running Kafka Stream Applications on AWS. We've seen issues with state locking, memory spiking to 100% and the instance dying, very slow startup on pulling down initial rocksdbs and so on Today we realised that the instances were expe

Re: Kafka Streams and AWS IOPS Credits

Thanks for reporting back Ian. Very useful information. Cheers, Damian On Wed, 24 May 2017 at 10:40 Ian Duffy wrote: > Hi All, > > In the past, we experienced lots of problems with running Kafka Stream > Applications on AWS. > > We've seen issues with state locking, memory spiking to 100% and t

Kafka Lag Monitor

Hi All, In the past my team was using burrow by linkedin for monitoring consumer group lag. We discovered it crashed a lot and we found ourselves constantly restarting it. As a fix, a co-worker has created his own solution that I would like to share with you https://github.com/zalando-incubator/r

Kafka Read Data from All Partition Using Key or Timestamp

Hi All , We have been using Kafka for our Use Case which helps in delivering real time raw logs.. I have a requirement to fetch data from Kafka by using offset .. DataSet Example : {"access_date":"2017-05-24 13:57:45.044","format":"json","start":"1490296463.031"} {"access_date":"2017-05-24 13:57:

Re: Kafka Authorization and ACLs Broken

Raghav/Darshan, Can you try these steps on a clean installation of Kafka? It works for me, so hopefully it will work for you. And then you can adapt to your scenario. *Create keystores and truststores:* keytool -genkey -alias kafka -keystore server.keystore.jks -dname "CN=KafkaBroker,O=Pivotal,C

Re: Kafka Authorization and ACLs Broken

Rajini I will try and report to you shortly. Many thanks. Raghav On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram wrote: > Raghav/Darshan, > > Can you try these steps on a clean installation of Kafka? It works for me, > so hopefully it will work for you. And then you can adapt to your scenario.

Re: Streams error handling

Watching it with interest thanks Not sure where appropriate to add suggestions but I'd vote for exceptions being passed along the stream in something like a hidden Either wrapper. Most of the KStream methods would ignore this but new onException() or similar methods would be able to examine the er

Re: Kafka Authorization and ACLs Broken

Hi Rajini Thank you very much. It perfectly works. I think in my setup I was trying to use a CA (certificate authority) to sign the certificates from client and server, and then adding it to trust store and keystore. I think in that process, I may have messed something. I will try above config wi

Loss of Messages

Hi, I am running Kafka as a 2 node cluster. When I am scaling up and down 1 kafka broker I am experiencing loss of messages at consumer end during reassignment of partitions. Do you know what might be the cause or how we can fix this issue at our end? I tried to search about this issue but couldn'

Re: Data loss after a Kafka Broker restart scenario

Hi, Fathima, Yes, the most efficient way to verify if a message is sent successfully is through the producer callback. You can take a look at PrintInfoCallback in org.apache.kafka.toolsVerifiableProducer as an example. Our system tests use that to verify if any data loss has occurred. Thanks, Ju

How to stop the producer if I use the incorrect bootstrap.servers

Hello, I am new to kafka and I found that if I use the incorrect bootstrap.server in producer, the kafka client seems that never give up althought the console show the exception below but no exception throws: Connection with /10.201.0.119 disconnected java.net.ConnectException: Conn

Re: Streams error handling

Thanks Mike, Most of the JIRAs there are internal cleanups, but for the user-facing ones we're planning on creating a wiki and collecting feedback like yours, and a KIP, so stay tuned (your current feedback already noted, thanks). Eno > On 24 May 2017, at 15:35, Mike Gould wrote: > > Watching

How to list all consumer-groups & their offset using confluent-kafka-python or librdkafka?

Hi, all We have upgraded Kafka from 0.8.2 to 0.10.2. In Kafka 0.8.2, we store offset for each consumer-group in Zookeeper. We used Python API to get offset for each consumer-group in ZK to monitor the lag for each Topic. After upgraded to Kafka 0.10.2, we plan to store consumer-group offset in Ka

Re: Loss of Messages

What is your in-sync timeout set to? -David On 5/24/17, 5:57 AM, "Vinayak Sharma" wrote: Hi, I am running Kafka as a 2 node cluster. When I am scaling up and down 1 kafka broker I am experiencing loss of messages at consumer end during reassignment of partitions. Do you

Upgrade from 0.8.2-beta to 0.10.2.1 -- Configuration changes

I need some help regarding kafka. Im moving from 0.8.2-beta to 0.10.2.1, but I cannot make the mirrormakers to come up. We are using same configs from old version on the newone, here is the example of the config files: *For Producer* # Producer # hkg1 zookeeper.connect=localhost:2181 metadata.b

Differece observed during kafka-producer-perf-test and JUnit producer test

Hi, I ran a JUnit test where i delivered a payload of ~5MB file to a Kafka cluser (3 broker, 3 zookeeper). These are all setup on my laptop. My test config for producer is the following: max.request.size=5242880 > batch.size=8192 > key.serializer=org.apache.kafka.common.serialization.StringSerial

Re: Kafka Authorization and ACLs Broken

I'm also having issues getting acls to work. Out of intereat, are you starting ur brokers with a jaas file, if so do u mind sharing the client and server side jaas entries so I can validate what I'm doing. mike marzo 908 209-4484 On May 24, 2017 10:54 AM, "Raghav" wrote: > Hi Rajini > > Thank

Disks full after upgrading kafka version : 0.8.1.1 to 0.10.0.0

In 24 hours the brokers started getting killed due to disk full. The retention period is 48 hrs and with 0.8 disks used to fill ~65% What is going wrong here ? This is production system. I am reducing the retention for the time being to 24 hrs.

Re: Kafka Streams and AWS IOPS Credits

Thanks for sharing Ian! Guozhang On Wed, May 24, 2017 at 3:49 AM, Damian Guy wrote: > Thanks for reporting back Ian. Very useful information. > > Cheers, > Damian > > On Wed, 24 May 2017 at 10:40 Ian Duffy wrote: > > > Hi All, > > > > In the past, we experienced lots of problems with running K

SASL and SSL

Hello! I'm not very clear on the behavior that we should expect when we configure Kafka to use the protocol SASL_SSL. Is SASL or SSL mutually exclusive here or can I authenticate with SASL and use SSL for encryption? If the latter is true, then is it correct to assume that encryption will take p

Re: Kafka Authorization and ACLs Broken

Mike I am not using jaas file. I literally took the config Rajini gave in the previous email and it worked for me. I am using ssl Kafka with ACLs. I am not suing kerberos. Thanks. On Wed, May 24, 2017 at 11:29 AM, Mike Marzo wrote: > I'm also having issues getting acls to work. Out of interea

Re: Kafka Authorization and ACLs Broken

Thanks. We will try it. Struggling with krb5 and acls mike marzo 908 209-4484 On May 24, 2017 9:29 PM, "Raghav" wrote: > Mike > > I am not using jaas file. I literally took the config Rajini gave in the > previous email and it worked for me. I am using ssl Kafka with ACLs. I am > not suing ke

Re: Kafka Authorization and ACLs Broken

I initially tried kerberos, but it felt too complicated, so gave up and only tried SSL. On Wed, May 24, 2017 at 7:47 PM, Mike Marzo wrote: > Thanks. We will try it. Struggling with krb5 and acls > > mike marzo > 908 209-4484 <(908)%20209-4484> > > On May 24, 2017 9:29 PM, "Raghav" wrote: > >>

Re: SASL and SSL

For completion, I saw Ismael Juma post an answer which contains the information I was looking for: http://comments.gmane.org/gmane.comp.apache.kafka.user/15140 SASL_SSL -> authentication using SASL AND connection is encrypted using SSL. On Wed, May 24, 2017 at 7:37 PM, Waleed Fateem wrote: > H

Kafka Configuration Question

Hello, I'm hoping someone could provide me with some assistance please. I am in the process of attempting to standing up a Kafka cluster and I have 7 nodes all of which has kafka and zookeeper installed. I have attached my server.properties file to verify whether or not I have anything misconfi