Adding replicas to existing topic cause data loss in some partitions

I have a kafka cluster, every topic in it has only one replica. Recently I extend every topic with 2 replicas. Most topics work fine, but some large topics have some problems with part of partitions. Consumer throw offset OutOfRange exception, the fact is consumer request offset is bigger than the

Re: Announcing Confluent

Best of Luck..keep rocking... On Fri, Nov 7, 2014 at 1:30 AM, Joe Brown wrote: > Best of luck!!! > > J > > On 6 Nov 2014, at 18:28, Jay Kreps wrote: > > > Hey all, > > > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > > company around Kafka called Confluent. We ar

Re: Cannot connect to Kafka from outside of EC2

Sameer, The server logs do not contain any non-INFO logs, which is a bit wired. Did you deploy the current trunk of Kafka? Also could you enable DEBUG level logging on Kafka brokers? Guozhang On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami wrote: > The server.log was taken separately. > We ran the

Re: consumer ack for high-level consumer?

0. Yes, if consumer crashed before commit its offset it can cause duplicates. 1. Yes, since from the consumer client's point of view, once the message is returned from the iterator it is considered as "consumed"; if you want consumer to only consider a message as consumed when it is processed by t

Re: corrupt recovery checkpoint file issue....

filed: https://issues.apache.org/jira/browse/KAFKA-1758 On Thu, Nov 6, 2014 at 11:50 PM, Jason Rosenberg wrote: > I'm still not sure what caused the reboot of the system (but yes it > appears to have crashed hard). The file system is xfs, on CentOs linux. > I'm not yet sure, but I think also be

Re: corrupt recovery checkpoint file issue....

I'm still not sure what caused the reboot of the system (but yes it appears to have crashed hard). The file system is xfs, on CentOs linux. I'm not yet sure, but I think also before the crash, the system might have become wedged. It appears the corrupt recovery files actually contained all zero

Re: nulls found in topic, created by recovery?

The topic is not compressed. The consumer used our fork of the python lib, which I had to modify to get over the nulls. -neil On Thu, Nov 6, 2014 at 2:16 PM, Neha Narkhede wrote: > IIRC, the bug that introduced the nulls was related to compressed data. Is > this topic compressed? Did you try to

Re: corrupt message

I've got almost the same error, but also haven't figured out the reason. 2014-11-07 9:05 GMT+08:00 Neha Narkhede : > This may be due to a bug in the client. Non-java Kafka clients are > maintained by the individual client owners. You might want to ping the > owner of your library directly. > > On

Re: consumer ack for high-level consumer?

Hi, Thanks for your response. Therefore, offsets in ZK may be out-of-date. It is possible to deliver duplicated messages when clients restart. I also wonder the possibilities of losing message. Is it possible that things occur in this order? 1. Client calls ConsumerIterator$next() to get a me

OffsetOutOfRange errors

Hello, I understand what this error means, just not sure why I keep running into it after 24-48 hrs of running fine consuming > 300 messages / second. What happens when a kafka log rolls over and some old records are aged out? I mean what happens to the offsets? We are using a python client w

Re: corrupt message

This may be due to a bug in the client. Non-java Kafka clients are maintained by the individual client owners. You might want to ping the owner of your library directly. On Mon, Nov 3, 2014 at 7:21 AM, Fredrik S Loekke wrote: > When running a C# producer against a kafka 0.8.1.1 server running o

Re: Kafka Cluster disaster decovery

A common solution for disaster recovery is to mirror the Kafka cluster into another one deployed in a separate data center. The mirroring is not synchronous so there might be some message loss when you lose the entire cluster in some disaster. Thanks, Neha On Mon, Nov 3, 2014 at 7:43 AM, Guozhang

Re: Issue with async producer

In 0.8.2 beta, we have a new java producer that's more performing. You can give it a try. Thanks, Jun On Thu, Nov 6, 2014 at 10:16 AM, Devendra Tagare < devendra.tag...@pubmatic.com> wrote: > Hi, > > Which version of Kafka are you using? > 0.8.0 > > Is the broker I/O or network saturated? - typ

Re: corrupt recovery checkpoint file issue....

I am also wondering how the corruption happened. The way that we update the OffsetCheckpoint file is to first write to a tmp file and flush the data. We then rename the tmp file to the final file. This is done to prevent corruption caused by a crash in the middle of the writes. In your case, was th

Re: Consumer lag keep increasing

Chen, Consumers lag either due to an I/O or network bottleneck or due to slow processing of messages by the user. To confirm that you are not hitting the latter issue, you can run a console consumer on the same data and observe the throughput that it provides and it's lag. Thanks, Neha On Wed, N

Re: kafka test jars in sbt?

The following is how samza references the kafka test jar in gradle. testCompile "org.apache.kafka:kafka_$scalaVersion:$kafkaVersion:test" Thanks, Jun On Thu, Nov 6, 2014 at 6:38 AM, Markus Jais wrote: > Hello, > > I want to use the kafka_2.10-0.8.2-beta-test.jar in my Scala project. > >

Re: Storing data in kafka keys

The keyed messages are typically used in two cases (1) you want messages with the same key to be in the same partition and therefore those messages with the same key will be consumed by the same consumer instance; (2) you want to enable the log compaction feature for retention such that the broker

Re: Producer and Consumer properties

For higher throughput, you want to configure the producer with a higher batch size. You may also want to enable compression. Thanks, Jun On Wed, Nov 5, 2014 at 6:46 AM, Eduardo Costa Alfaia wrote: > Hi Dudes, > > I would like to know if the producer and consumer’s properties files into > the c

Re: Location of Logging Files/How To Turn On Logging For Kafka Components

The log4j entries before that error should tell you the cause of the error. Thanks, Jun On Tue, Nov 4, 2014 at 11:25 PM, Alex Melville wrote: > Background: > > I have searched for a while online, and through the files located in the > kafka/logs directory, trying to find where kafka writes log

Re: High CPU usage of Crc32 on Kafka broker

I suspect it is possible to save and reuse the CRCs though it might be a bit of an invasive change. I suspect the first usage is when we are checking the validity of the messages and the second is from when we rebuild the compressed message set (I'm assuming you guys are using compression because I

Re: Kafka Release timelines

0.8.2 should be available in a month. Though 0.9 might take a couple more months and there is a good chance that it will not be this year. Thanks, Neha On Thu, Nov 6, 2014 at 3:01 AM, dinesh kumar wrote: > Hi, > I found the future release plan wiki here >

Re: High CPU usage of Crc32 on Kafka broker

Sure. Here is the link to the screen shot of jmc with the JTR file loaded: http://picpaste.com/fligh-recorder-crc.png On Thu, Nov 6, 2014 at 2:12 PM, Neha Narkhede wrote: > Allen, > > Apache mailing lists don't allow attachments. Could you please link to a > pastebin or something? > > Thanks,

Re: Interaction of retention settings for broker and topic plus partitions

To clarify though, is it correct that a per topic limit will always override the default limit of the same type? (e.g. a large per-topic retention hours vs. a small default retention hours)? That's correct. On Thu, Nov 6, 2014 at 9:34 AM, Jason Rosenberg wrote: > Jun, > > To clarify though, is

Re: nulls found in topic, created by recovery?

IIRC, the bug that introduced the nulls was related to compressed data. Is this topic compressed? Did you try to run a consumer through the topic's data or alternately the DumpLogSegments tool? On Thu, Nov 6, 2014 at 12:56 PM, Neil Harkins wrote: > Hi all. I saw something weird yesterday on our

Re: High CPU usage of Crc32 on Kafka broker

Allen, Apache mailing lists don't allow attachments. Could you please link to a pastebin or something? Thanks, Neha On Thu, Nov 6, 2014 at 12:02 PM, Allen Wang wrote: > After digging more into the stack trace got from flight recorder (which is > attached), it seems that Kafka (0.8.1.1) can opt

Re: Strategies for high-concurrency consumers

On Nov 6, 2014, at 11:43 AM, Neha Narkhede wrote: > Zookeeper is likely the bottleneck if rebalancing takes a very long time. > As Jay said, this will be addressed in the consumer rewrite planned for > 0.9. Few more workarounds that were tried at LinkedIn - 1) To deploy > Zookeeper on SSDs and 2)

nulls found in topic, created by recovery?

Hi all. I saw something weird yesterday on our "leaf" instances which run kafka 0.7.2 (and mirror to kafka 0.8 via our custom code). I fully realize everyone's instinctual response is "upgrade, already.", but I'd like to have an internals discussion to better understand what happened, as I suspect

Re: Announcing Confluent

Best of luck!!! J On 6 Nov 2014, at 18:28, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time data platform we built out at LinkedIn.

Re: High CPU usage of Crc32 on Kafka broker

After digging more into the stack trace got from flight recorder (which is attached), it seems that Kafka (0.8.1.1) can optimize the usage of Crc32. The stack trace shows that Crc32 is invoked twice from Log.append(). First is from the line number 231: val appendInfo = analyzeAndValidateMessageSet

Re: Announcing Confluent

Jay, Neha and Jun congratz!! On Thu, Nov 6, 2014 at 11:09 AM, Bhavesh Mistry wrote: > HI Guys, > > Thanks for your awesome support. I wish you good luck !! Thanks for open > sources Kafka !! > > Thanks, > > Bhavesh > > On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango > wrote: > > > Congrats

Re: Strategies for high-concurrency consumers

Jack, Zookeeper is likely the bottleneck if rebalancing takes a very long time. As Jay said, this will be addressed in the consumer rewrite planned for 0.9. Few more workarounds that were tried at LinkedIn - 1) To deploy Zookeeper on SSDs and 2) Turning sync on every write off (zookeeper.forceSync

Re: Strategies for high-concurrency consumers

Unfortunately the performance of the consumer balancing scales poorly with the number of partitions. This is one of the things the consumer rewrite project is meant to address, however that is not complete yet. A reasonable workaround may be to decouple your application parallelism from the number

Re: Announcing Confluent

HI Guys, Thanks for your awesome support. I wish you good luck !! Thanks for open sources Kafka !! Thanks, Bhavesh On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango wrote: > Congrats. Wish you all the very best and success. > > Thanks, > Raja. > > On Thu, Nov 6, 2014 at 1:36 PM, Niek Sande

Re: No longer supporting Java 6, if? when?

Yeah it is a little bit silly that people are still using Java 6. I guess this is a tradeoff--being more conservative in our java support means more people can use our software, whereas upgrading gives us developers a better experience since we aren't stuck with ancient stuff. Nonetheless I would

Strategies for high-concurrency consumers

Hi all, We are building a system that will carry a high volume of traffic (on the order of 2 billion messages in each batch), which we need to process at a rate of 50,000 messages per second. We need to guarantee at-least-once delivery for each message. The system we are feeding has a latency o

Re: No longer supporting Java 6, if? when?

Java6 is supported on CDH4 but not CDH5. On Thu, Nov 6, 2014 at 9:54 AM, Koert Kuipers wrote: > when is java 6 dropped by the hadoop distros? > > i am still aware of many clusters that are java 6 only at the moment. > > > > On Thu, Nov 6, 2014 at 12:44 PM, Gwen Shapira > wrote: > > > +1 for dro

Re: Announcing Confluent

Congrats. Wish you all the very best and success. Thanks, Raja. On Thu, Nov 6, 2014 at 1:36 PM, Niek Sanders wrote: > Congrats! > > On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > > Hey all, > > > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > > company around K

Re:Announcing Confluent

Congratulations Jay, Jun and Neha. Great news. Wish you good luck!!! - Original Message - From: users@kafka.apache.org To: d...@kafka.apache.org, users@kafka.apache.org At: Nov 6 2014 13:28:51 Hey all, I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a company around

Re: Announcing Confluent

Congrats! On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time data platform we built out at LinkedIn. We

Re: Announcing Confluent

Awesome. Congrats to all of you! On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time data platform we b

Re: Announcing Confluent

Congrats Jay, Neha and Jun. Look forward to it. On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time dat

Announcing Confluent

Hey all, I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a company around Kafka called Confluent. We are planning on productizing the kind of Kafka-based real-time data platform we built out at LinkedIn. We are doing this because we think this is a really powerful idea and we

RE: Issue with async producer

Hi, Which version of Kafka are you using? 0.8.0 Is the broker I/O or network saturated? - typical writes are around 45 MB per broker & there are streaming as well as batch consumers. We are already sending across compressed packets & the batch size is 100. Also, we tried the sync producer but

Re: "metric.reporters" is not working

I added the following ugly reflection code and it's working try { Field f = ProducerConfig.class.getDeclaredField("config"); f.setAccessible(true); ConfigDef config = (ConfigDef) f.get(ConfigDef.class); config.define(ServoReporter.class.getName(), Co

Re: Consumer and Producer configs

Hello Eduardo, If you are using console producer / consumer, you can set the configs in command line starting them; if you are wiring the clients directly, then you can set them in a Properties and pass them to the constructor. Guozhang On Thu, Nov 6, 2014 at 7:10 AM, Eduardo Costa Alfaia wrote

Re: No longer supporting Java 6, if? when?

when is java 6 dropped by the hadoop distros? i am still aware of many clusters that are java 6 only at the moment. On Thu, Nov 6, 2014 at 12:44 PM, Gwen Shapira wrote: > +1 for dropping Java 6 > > On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker < > sschlans...@opentable.com > > wrote: > >

Re: corrupt recovery checkpoint file issue....

Jason, Yes I agree with you. We should handle this more gracefully as the checkpoint file dump is not guaranteed atomic. Could you file a JIRA? Guozhang On Thu, Nov 6, 2014 at 6:31 AM, Jason Rosenberg wrote: > Hi, > > We recently had a kafka node go down suddenly. When it came back up, it > ap

Re: Spark and Kafka

This is my window: reduceByKeyAndWindow( new Function2() { @Override public Integer call(Integer i1, Integer i2) { return i1 + i2; } }, new Function2() { public Integer call(Integer i1, Integer i2) { return i1 - i2; } }, new Duration(60

Re: consumer ack for high-level consumer?

That is correct. Guozhang On Wed, Nov 5, 2014 at 9:18 PM, Chia-Chun Shih wrote: > Hi, > > Thanks for your response. I just read source code and found that: > > 1) ConsumerIterator$next() use PartitionTopicInfo$resetConsumeOffset to > update offsets in PartitionTopicInfo objects. > 2) Zookee

Re: No longer supporting Java 6, if? when?

+1 for dropping Java 6 On Thu, Nov 6, 2014 at 9:31 AM, Steven Schlansker wrote: > Java 6 has been End of Life since Feb 2013. > Java 7 (and 8, but unfortunately that's too new still) has very compelling > features which can make development a lot easier. > > The sooner more projects drop Java 6

Re: OffsetOutOfRange Error

Jimmy, I am not very familiar with the python java client, you may directly ask its author: https://cwiki.apache.org/confluence/display/KAFKA/Clients On Thu, Nov 6, 2014 at 7:57 AM, Jimmy John wrote: > I dug deeper and saw this during normal operation: > > In the kafka broker log: > > [2014-11

Re: Spark and Kafka

What's the window size? If the window is around 10 seconds and you are sending data at very stable rate, this is expected. On Thu, Nov 6, 2014 at 9:32 AM, Eduardo Costa Alfaia wrote: > Hi Guys, > > I am doing some tests with Spark Streaming and Kafka, but I have seen > something strange, I hav

Spark and Kafka

Hi Guys, I am doing some tests with Spark Streaming and Kafka, but I have seen something strange, I have modified the JavaKafkaWordCount to use ReducebyKeyandWindow and to print in the screen the accumulated numbers of the words, in the beginning spark works very well in each interaction the nu

Re: Interaction of retention settings for broker and topic plus partitions

Jun, To clarify though, is it correct that a per topic limit will always override the default limit of the same type? (e.g. a large per-topic retention hours vs. a small default retention hours)? Jason On Sat, Sep 20, 2014 at 12:28 AM, Jun Rao wrote: > That's right. The rule is that a log seg

Re: No longer supporting Java 6, if? when?

Java 6 has been End of Life since Feb 2013. Java 7 (and 8, but unfortunately that's too new still) has very compelling features which can make development a lot easier. The sooner more projects drop Java 6 the better, in my opinion :) On Nov 5, 2014, at 7:45 PM, Worthy LaFollette wrote: > Mostl

Re: Disactivating Yammer Metrics Monitoring

Hi Francois, We had the exact same problem. We embed Kafka in our service container, and we use yammer metrics to see data about the whole app (e.g. kafka, the jvm, the service container wrapping it). However, as you observed, by default, kafka produces an insane amount of metrics. So what we d

Re: zookeeper upgrade or remove zookeeper dependency

We have been using zk 3.4.6 (and we use curator), without any problems with kafka, for quite a while now Jason On Thu, Sep 18, 2014 at 2:18 PM, Mingtao Zhang wrote: > Great :) > > Best Regards, > Mingtao > > On Thu, Sep 18, 2014 at 2:04 PM, Guozhang Wang wrote: > > > Hi Mingtao, > > > > We

Re: corrupt recovery checkpoint file issue....

forgot to mention, we are using 0.8.1.1 Jason On Thu, Nov 6, 2014 at 9:31 AM, Jason Rosenberg wrote: > Hi, > > We recently had a kafka node go down suddenly. When it came back up, it > apparently had a corrupt recovery file, and refused to startup: > > 2014-11-06 08:17:19,299 WARN [main] s

OffsetOutOfRange Error

I dug deeper and saw this during normal operation: In the kafka broker log: [2014-11-03 21:39:25,658] ERROR [KafkaApi-8] Error when processing fetch request for partition [activity.stream,5] offset 7475239 from consumer with correlation id 69 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRange

Re: OffsetOutOfRange Error

I dug deeper and saw this during normal operation: In the kafka broker log: [2014-11-03 21:39:25,658] ERROR [KafkaApi-8] Error when processing fetch request for partition [activity.stream,5] offset 7475239 from consumer with correlation id 69 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRange

Consumer and Producer configs

Hi Guys, How could I use the Consumer and Producer configs in my Kafka environment? Thanks -- Informativa sulla Privacy: http://www.unibs.it/node/8155

kafka test jars in sbt?

Hello, I want to use the kafka_2.10-0.8.2-beta-test.jar in my Scala project. It can be found here: http://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.10/0.8.1.1/ In my build.sbt I write the following definition: "org.apache.kafka" % "kafka_2.10" % "0.8.2-beta-test" But sbt cannot f

corrupt recovery checkpoint file issue....

Hi, We recently had a kafka node go down suddenly. When it came back up, it apparently had a corrupt recovery file, and refused to startup: 2014-11-06 08:17:19,299 WARN [main] server.KafkaServer - Error starting up KafkaServer java.lang.NumberFormatException: For input string: "^@^@^@^@^@^@^@^@^

Re: Information

Hi, These are GC logs. Specially these are young GC collection logs. They look normal to me. Are you observing any GC pauses on kafka server? pl check below link. https://kafka.apache.org/documentation.html#java Regards, Kumar On Thu, Nov 6, 2014 at 4:53 PM, Eduardo Costa Alfaia wrote

Information

Hi Guys Anyone could explain me this information? 208K), 0.0086120 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 2014-11-06T12:20:55.673+0100: 1256.382: [GC2014-11-06T12:20:55.674+0100: 1256.382: [ParNew: 551115K->2816K(613440K), 0.0204130 secs] 560218K->13933K(4126208K), 0.0205130 secs] [

Kafka Release timelines

Hi, I found the future release plan wiki here . I see that 0.8.2 is still in beta even though it was stated for September. What is the expected date for 0.9 release? Thanks, Dinesh

Re: Error using migrationtool for upgrading 0.7 to 0.8

Thanks to this last hint we finally got it to work! One problem was that the new kafka 0.8 cluster, was compiled with scala 2.10, while the old 0.7 one was compiled with scala 2.8. As the MigrationTool uses 0.8 binaries, but also libraries from 0.7, there was a mismatch and it wasn't finding the c