Why multi zookeeper show different topic count

I deploy 3 Zookeeper instances on hd-kafka-01/02/03 3 Kafka server instances on hd-kafka-03/04/05 I created 4 topics earlier when I'm trying to list topics bin/kafka-topics.sh --list --zookeeper hd-kafka-03:2181,hd-kafka-0 3:2181,hd-kafka-05:2181 I get different results randomly: 5 topics or 1

Kafka OOME: Direct buffer memory

We are running kafka 0.9.0.1 in production and saw these exceptions: [2016-06-23 22:55:10,239] INFO [KafkaApi-3] Closing connection due to error during produce request with correlation id 6 from client id kafka-python with ack=0 Topic and partition to exceptions: [xyx,8] -> kafka.common.Mes

Re: documenting Kafka Streams PageView examples

Phil, I suggest to ask this question in the Confluent Platform mailing list because you're referring to code under https://github.com/confluentinc/examples (i.e. code that is not part of the Apache Kafka project). Best, Michael On Tue, Jul 5, 2016 at 5:34 PM, Philippe Derome wrote: > Would an

Re: documenting Kafka Streams PageView examples

Correction: Just realized that I misread your message. You are indeed referring to the code examples in Apache Kafka. ;-) On Wed, Jul 6, 2016 at 11:35 AM, Michael Noll wrote: > Phil, > > I suggest to ask this question in the Confluent Platform mailing list > because you're referring to code un

Re: Question on partitions while consuming multiple topics

Snehal beat me to it, as my suggestion would have also been to take a look at Kafka Streams. :-) Kafka Streams should be the easiest way to achieve what you're describing. Snehal's links are good starting points. Further pointers are: https://github.com/confluentinc/examples/blob/master/kafka-s

Re: Question on partitions while consuming multiple topics

PS: The previous example links that I shared are for the latest `trunk` version of Kafka. If you want to use the latest official release instead (Kafka 0.10.0.0), which most probably is what you want, then please use the following links to these examples. Note the `kafka-0.10.0.0-cp-3.0.0` branc

Opportunity to contribute in Apache Kafka

Hello, We are a group of students that are currently pursuing our undergraduate degree in Computer Science from Pune Institute of Computer Technology(PICT), Maharashtra, India. We will be graduating in June 2017 and are currently in our final year. For our B.E project, we have decided to work on D

Re: Opportunity to contribute in Apache Kafka

Hi Shubham and team, Considering development time,below should be good . Name: Creating Kafka connector/s 1. Read MySQL binlog and send data to kafka in JSON/Avro. 2. Read MongoDB oplog and send data to Kafka in JSON/Avro. To start with simple, you can consider single DB node and then add comple

Re: documenting Kafka Streams PageView examples

yes, it's a very similar example and I am interested in the Kafka one for the serialization aspect of it, which is a bit richer than on Confluent's... On Wed, Jul 6, 2016 at 5:35 AM, Michael Noll wrote: > Correction: Just realized that I misread your message. You are indeed > referring to the

Lots of messages when starting Kafka

Hi, this is my first time using a mailing list so please forgive me if I'm doing something wrong. When I start Kafka, I get a ton of messages like this: WARN Found a corrupted index file, /kafka/logs/__consumer_offsets-48/.index, deleting and rebuilding index... (kafka.l

Re: Lots of messages when starting Kafka

If you kill Kafka with SIGTERM, then naturally it will leave behind corrupt index files. It will safely rebuild them. Don't worry. The Group Metadata Manager messages are normal -- that's just Kafka starting up. Perhaps the corrupted index log messages could be less scary, though. This is a very

Re: Lots of messages when starting Kafka

>If you kill Kafka with SIGTERM, then naturally it will leave behind corrupt index files. It will safely rebuild them. Don't worry. I think you mean SIGKILL. SIGTERM should be handled normally an won't result in corrupted index files at startup. __ Sam Pegler Production Engineer T. +44(0) 07

Re: Lots of messages when starting Kafka

You're right Sam. Getting my signals confused before coffee. Fabian, how did you in fact kill Kafka? "kill"? Ctrl+C? -Peter On Wed, Jul 6, 2016 at 07:00 Sam Pegler wrote: > >If you kill Kafka with SIGTERM, then naturally it will leave behind > corrupt > index files. It will safely rebuild them

Re: Lots of messages when starting Kafka

I use "docker stop" which sends SIGTERM and then waits 10 seconds before it sends SIGKILL. I assume it doesn't get to send SIGKILL because when I don't use Docker, a usual Ctrl+C stops Kafka way faster than 10 seconds. On 06/07/16 16:05, Peter Davis wrote: You're right Sam. Getting my signals

Re: Lots of messages when starting Kafka

Thank you! If there's nothing to be afraid of, I will ignore these messages from now on. On 06/07/16 15:54, Peter Davis wrote: If you kill Kafka with SIGTERM, then naturally it will leave behind corrupt index files. It will safely rebuild them. Don't worry. The Group Metadata Manager messages

Re: Lots of messages when starting Kafka

I've seen Kafka take much longer than 10 seconds to stop normally. (I don't use docker.) controlled.shutdown.enable (true) Enable controlled shutdown of the server controlled.shutdown.max.retries (3) Controlled shutdown can fail for multiple reasons. This determines the number of retries when su

Kafka does not preserve an offset on topic.

Hello, I tried every different property I can think of - I have set ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG = true; ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG = 1000, but I have also tried with many different values And still every time I start my consumer - I am receiving all the messages

Kafka - offset preservation

Hello, I tried every different property I can think of - I have set ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG = true; ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG = 1000, but I have also tried with many different values And still every time I start my consumer - I am receiving all the messages

Last offset in all partitions

Hi Is there a way to get the last offset written by all partitions of a topic programmatically using the 0.10.0.0 API? At the moment I use KafkaConsumer.seekToEnd as seen in this gist[1] but maybe there is a better, more efficient, way to do it? Cheers, -Kristoffer [1] https://gist.github.com/k

Re: Last offset in all partitions

We do this through our monitoring agents by pulling it as a metric from the LogEndOffset beans. By putting it into our metrics system we get a mapping of timestamp to offset for every partition with (currently) 60 second granularity. Useful for offset resets and other tasks. -Todd On Wednesday, J

Monitoring offset lag

I recently had a problem on my production which I believe was a manifestation of the issue kafka-2978 (Topic partition is not sometimes consumed after rebalancing of consumer group), this is fixed in 0.9.0.1 and we will upgrade our client soon. However, it made me realise that I didn’t have any

Re: Heartbeating during long processing times

Shikhar, Thanks for pointing me to KIP-62. Once implemented, it will make workers that take a long time processing messages a lot simpler to implement. Until then, we have to continue using the pause/poll/resume pattern. That said, as fares I can tell, this pattern has not been well documented.

Re: Monitoring offset lag

Yeah, I've written dissertations at this point on why MaxLag is flawed. We also used to use the offset checker tool, and later something similar that was a little easier to slot into our monitoring systems. Problems with all of these is why I wrote Burrow (https://github.com/linkedin/Burrow) For m

Now available - kafka-tools from LinkedIn SRE

I’ve mentioned this one a couple times in a talk here or a conversation there, but we hadn’t officially announced a release yet. Now that we’ve spent some time refactoring the initial code and adding tests, I’m happy to announce the release of our kafka-tools repository. The kafka-tools repository

Find partition offsets in a kerberized kafka cluster

Hi, Does the kafka.tools.GetOffsetShell utility work with a kerberized kafka cluster ? I doubt that it uses the old consumer which does not work with kerberos and hence cannot be used with kerberos. Is there a utlity that has this functionality in a kerberized cluster ? I currently do this by t

Re: Last offset in all partitions

That's awesome! I can see the JMX bean [1] in our current 0.8.2 brokers and the number seems updated in real time per partition? I cannot find any official documentation on it? How safe is it to use? Also we're planning on upgrading to 0.10.0.0 - is it still available? Cheers, -Kristoffer [1] ka

RE: Last offset in all partitions

Checkout the Consumer API http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html and search for the method "seekToEnd" Here's the "text" from the API Doc - seekToEnd public void seekToEnd(Collection partitions) Seek to the last offset for each of

Re: Last offset in all partitions

Yeah, it reflects the last offset in each partition all the time. The one thing to be careful of is that it can be emitted from any replica (whichever is the leader). So our tools that use the information always look at all brokers and then take the larger non-null value. We've been using it for a

Re: Last offset in all partitions

Thanks for the heads up and information Todd! Cheers, -Kristoffer On Wed, Jul 6, 2016 at 8:39 PM, Todd Palino wrote: > Yeah, it reflects the last offset in each partition all the time. The one > thing to be careful of is that it can be emitted from any replica > (whichever is the leader). So our

Subscribe to kafka user email group.

Thanks Manoj

Re: Subscribe to kafka user email group.

You need to send an email to users-subscr...@kafka.apache.org See https://kafka.apache.org/contact.html -Matthias On 07/06/2016 10:26 PM, Manoj Chaudhary wrote: > Thanks > Manoj > signature.asc Description: OpenPGP digital signature

Kafka MirrorMaker and Data encryption

Hi, We have local kafka clusters in both us-east and us-west data centers in AWS. We want to mirror the data in the other cluster so each region has all the data. If we use Kafka MirrorMaker, the remote consumer will be consuming data over the internet. Is there a way to encrypt this data without

Re: Kafka MirrorMaker and Data encryption

We're not doing it in AWS, but we are doing it for other flows. The key to not having performance problems is to not consume over SSL. This means turning the previously recommended architecture around, and having the mirror makers located in the region that you are consuming from. This way you can

Re: documenting Kafka Streams PageView examples

The typed version of the example (PageViewTypedDemo) is what represents some difficulty for someone new to Kafka (or CP3). *I think it would be easier/quicker to complete the documentation of that example than to answer my questions below* I am reusing the same JsonPOJOSerializer and JsonPOJOD

Re: documenting Kafka Streams PageView examples

I think I should simply follow Kafka The Definitive Guide Chapter 3 for a good Avro producer example instead. It does not introduce some Jackson JSON layer and still provides the type safety using POJO generated classes from Avro. On Wed, Jul 6, 2016 at 9:20 PM, Philippe Derome wrote: > The type

Kafka Streams : Old Producer

Can kafka streams library work with the messages produced by 0.9.0.1 producer? I guess not since the old producer would not add timestamp. ( I am getting invalid timestamp exception) As I cannot change our producer application setup, I have to use 0.9.0.1 producer. Is there a workaround that I can

Re: Expired messages in kafka topic

Hi, I am trying to set up a kafka dev environment for the past few days. I followed this link; the only difference being that I downloaded the latest scala IDE, instead of Eclipse. Any pointers to resolv

Pros and cons of dockerizing kafka brokers?

Hi, I am currently testing a custom docker volume driver plugin for AWS EFS/EBS access and mounting. So, running kafka broker inside a container makes will ease up a lot of configuration issues wrt storage for me. Are there any pros and cons of dockerizing kafka broker? Off the top of my head, sin