A Question about '*.index' file

Hi, all. I set a broker’s log directory to ‘/tmp/kafka-logs-1’, and found that there are .index, .log in the directory ‘/tmp/kafka-logs-1/test-0/’. Whenever the broker receives some messages, that messages are appended in the file “.lo

Re: A Question about '*.index' file

카푸카 어디에 쓰나요? 나의 iPhone에서 보냄 2013. 10. 31. 오후 5:50 Lee, Yeon Ok (이연옥) 작성: > Hi, all. > > I set a broker’s log directory to ‘/tmp/kafka-logs-1’, > and found that there are .index, .log > in the directory ‘/tmp/kafka-logs-1/test-0/’. > > Whenever the brok

Remote consumer

Hi, We are architecting an application in AWS with multi region deployment and were evaluating kafka as the messaging system for data replication and activity tracking. The initial idea is to have the producer and broker cluster to reside in Ireland. We wanted to know the feasibility of havi

Re: A Question about '*.index' file

The index file gets a new entry every index.interval.bytes. The index maps logical to physical data and is used to look up messages by logical offset. Thanks, Neha On Oct 31, 2013 1:50 AM, "Lee, Yeon Ok (이연옥)" wrote: > Hi, all. > > I set a broker’s log directory to ‘/tmp/kafka-logs-1’, > and fou

Re: Remote consumer

One option would be to use Cassandra for the multi data center replication and have Kafka Consumes update the Cassandra ring in each data center. This allows you to have active / active / active applications. It also allows you to choose a ring and map/reduce the data out of it (like with Pig) wh

Re: Topic related API

Thanks Viktor,but you may misunderstand what I mean,when I send a topicMetadataRequest for a none existed topic,Kafka server can auto create the topic ,but it take time before I can get the metadata of the new topic,so if look up zookeeper to make sure the topic is created.and my question still

Re: Roadmap

0.8 release is in progress and Joe Stein can give an update on when. What Maven dependency issues did you run into? Some of us have started working on 0.9 and the earliest likely release data is probably end of Jan 2014. I updated the versions page to reflect that. Thanks, Neha On Wed, Oct 30, 2

Purgatory

Hello, What is purgatory? I believe the following two properties relate to consumer and producer respectively. Could someone please explain the significance of these? fetch.purgatory.purge.interval.requests=100 producer.purgatory.purge.interval.requests=100 Thanks, Priya

Re: Questions about producer API

Thank you, Neha. I mainly wanted to understand if it was because of historic reasons or some fundamental reason. I think managing configuration will be simpler if both sides use ZooKeeper for discovery. Great to hear about the client rewrite project going on. Thanks to you and the other contrib

Re: Roadmap

Thanks Neha. I get the following compilation error in Eclipse, and in a maven build. The type scala.collection.Seq cannot be resolved. It is indirectly referenced from required .class files AppTest.java /kafkaesque/src/test/java/com/gridpoint/kafkaesque line 68 Java Problem org.apache.kafk

Re: Roadmap

Do you have Scala as a dependency in your pom? org.scala-lang scala-library 2.8.0 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Roadmap

I am testing all the artifacts for the 0.8.0 release currently (they are all built and uploaded). Hopefully done later today/tonight and will call a vote or raise issues found then. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Securi

unused topic gets deleted?

Hello, I have been playing with kafka producer and consumer. I have created different topics at different times with different replica and partition options. After 7 days of un-use, list-topics script does not list the unused topic. However, I can see the log and index files for such topic partit

Re: Kafka 0.8, Scala 2.10.2, OSGi

Thanks, got it to work, much appreciated. On 10/30/13, 5:23 PM, "Joe Stein" wrote: >you need a pgp keypair. > >if you use sbt you could try http://www.scala-sbt.org/sbt-pgp/usage.html > >or without sbt get OpenPGP installed http://linux.die.net/man/1/gpg2 and >then > >gpg2 --gen-key > > >On Wed,

Re: Kafka 0.8, Scala 2.10.2, OSGi

Ok, so I guess this means there isn’t explicit support for OSGi yet, I’ll take a look at the client rewrite when I get a chance, thanks. On 10/30/13, 5:31 PM, "Joe Stein" wrote: >In regards to OSGi maybe it is something you would propose and help out >with for the client rewrite >https://cwiki.a

Re: Why Apache Kafka is bettern than any other Message System?

I'll give you my take in case it helps. Kafka achieves great throughput + durability because 1) The broker does minimal work a) Routing is done by producers b) State management is done by consumers Other messaging brokers typically have to keep track of which messages have been dispatch

java.lang.NoClassDefFoundError: scala/reflect/ClassManifest

I built and deployed latest 0.8 version on Scala 2.10.2 to my local maven repository. I also started a sample producer API app using Scala 2.10.2, building the app causes java.lang.NoClassDefFoundError error when calling ProducerConfig, see below for code. package com.foo.bar package common i

Re: Roadmap

That helped :) I also had to add Log4J and Metrics, but I was then able to send a message to the broker. I'm revealing my Maven ignorance, but I'm unclear why I needed to add transitive dependencies. Also, since I will be writing Scala 2.10 as well as Java, I presume I will need the com.sksamuel.

Re: unused topic gets deleted?

Yes, this seems unexpected. Do you see an error on list-topics or does it return nothing? Also, could you file a JIRA for us to look into? Thanks, Neha On Thu, Oct 31, 2013 at 9:54 AM, Priya Matpadi wrote: > Hello, > > I have been playing with kafka producer and consumer. I have created > diffe

Re: Roadmap

A build with Scala 2.10 is published in the 0.8.0 release so you should use that once released. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Purgatory

Hi Priya my understanding is producer requests will be delayed (and put in request purgatory) only if your producer uses ack=-1. It will be in the purgatory (delayed) until all brokers have acknowledged the messages to be replicated. The documentation suggests to monitor the ProducerRequestPurgato

SimpleConsumer cannot read KeyedMessage.

Hi, I am following the https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example When I send KeyedMessage with StringEncoder, I can get the messages sent: for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(m_topic, m_partition)) { //handle messages } But whe

Re: Robust of cassandra

Hi, Neha Thanks for replying, I am currently using the 0.8 version. I did not set any replication factor, it keeps the default from package. When I check the leader for that topic, it is none. It is weird, There is no problem for any other topics except that topic that I am using. For now, I cannot

Re: java.lang.NoClassDefFoundError: scala/reflect/ClassManifest

Turns out that I forgot to change my POM to use 2.10 version, this fixes it in case someone else is trying to do the same thing org.apache.kafka kafka_2.10 0.8.0 Bob On 10/31/13, 10:53 AM, "Ngu, Bob" wrote: >I built and deployed latest 0.8 version on Scala 2.10.2 to my local maven

Re: SimpleConsumer cannot read KeyedMessage.

It seems the reason is I enabled gzip compression. what the code would like to consume compressed messages? thanks. On Thu, Oct 31, 2013 at 11:26 AM, Lu Xuechao wrote: > Hi, > > I am following the > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > > When I send

Re: Purgatory

Hello Priya, You can find the definitions of these two configs here: http://kafka.apache.org/documentation.html#brokerconfigs Guozhang On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe wrote: > Hi Priya > > my understanding is producer requests will be delayed (and put in request > purgatory) onl

Re: Topic related API

Hello, Have you tried the list topic tool? https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools Guozhang On Thu, Oct 31, 2013 at 8:53 AM, 小宇 wrote: > Thanks Viktor,but you may misunderstand what I mean,when I send a > topicMetadataRequest for a none existed topic,Kafka server

Re: SimpleConsumer cannot read KeyedMessage.

Xuechao, The same code should also work with gzip compression on KeyedMessage. Did you see any exceptions in the consumer logs? Guozhang On Thu, Oct 31, 2013 at 2:23 PM, Lu Xuechao wrote: > It seems the reason is I enabled gzip compression. > > what the code would like to consume compressed m

Using Kafka 0.8 from Scala and Akka

So I have the 0.8. Beta 1 consumer Java example running now. Is there a Scala API documented somewhere? What about Akka examples? RxJava? When I Google for that I get old links to using Kafka as a durable Akka mailbox. I don't think that's what I'm looking for. I just want something "reactive" tha

Re: Robust of cassandra

The default replication factor is 1, so even if you have one broker failure, you can run into a situation with no leader for some partitions. You probably want to use a higher replication factor. Thanks, Neha On Thu, Oct 31, 2013 at 12:07 PM, Jiang Jacky wrote: > Hi, Neha > Thanks for replying

Re: Using Kafka 0.8 from Scala and Akka

I am in the process of releasing out Scala and RxJava consumer(s) on github. Will be releasing it soon. Keep an eye out. On Thu, Oct 31, 2013 at 3:49 PM, Richard Rodseth wrote: > So I have the 0.8. Beta 1 consumer Java example running now. > > Is there a Scala API documented somewhere? What abo

Throughput Questions

Hi guys, I have some throughput questions. I try to test the throughput using both the High Level Consumer and Simple Consumer example from the document. But I get much lower throughput of simple consumer than the high level consumer. I run the test in the cluster and I'm sure I distribute the le

New error on on 0.72 Kafka brokers

We suddenly started seeing these messages from one our consumers tonight. What do they mean? Our high-level consumers -- most of them -- are not moving. What has happened? Philip 2013-11-01 01:58:09,033 [ERROR] [FetcherRunnable.error] error in FetcherRunnable for pro cessed:6-6: fetched offset =

Topic creation on restart

Some times during a rolling restart, I see lots of these messages during the restart (these happened in the logs of the 2nd server in the cluster to restart (broker 11), after having restarted started, and during the time the 3rd node (broker 9) is doing a controlled shutdown before stopping). Is

Re: Throughput Questions

I suspect this is something in your code as the high-level consumer uses the simple consumer. On Thu, Oct 31, 2013 at 5:33 PM, hsy...@gmail.com wrote: > Hi guys, I have some throughput questions. > > I try to test the throughput using both the High Level Consumer and Simple > Consumer example f

Re: New error on on 0.72 Kafka brokers

D'oh. Bad config on our part. Something we thought we had fixed long ago, but it crept back in. Make sure fetch sizes are big enough! Philp On Oct 31, 2013, at 7:18 PM, "Philip O'Toole" wrote: > We suddenly started seeing these messages from one our consumers tonight. > What do they mean?

Re: Connection reset by peer Error in Kafka server.log

That typically means the client closed the socket in the middle of a request. Are you using hardware load balancer? Thanks, Jun On Wed, Oct 30, 2013 at 9:03 PM, Banerjee, Aparup wrote: > Hi, > > I keep getting this error in Kafka server.log. I don't see anything in my > producer or consumer lo

Re: Connection reset by peer Error in Kafka server.log

No I am not using a LB. Infact I keep getting the same error even if my Kafka server and consumer are on same box. The client here is consumer btw. Aparup > On Oct 31, 2013, at 9:28 PM, "Jun Rao" wrote: > > That typically means the client closed the socket in the middle of a > request. Are you

Re: SimpleConsumer cannot read KeyedMessage.

Is that related to https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whybrokersdonotreceiveproducersentmessages%3F? Thanks, Jun On Thu, Oct 31, 2013 at 2:23 PM, Lu Xuechao wrote: > It seems the reason is I enabled gzip compression. > > what the code would like to consume compressed me

Re: Topic creation on restart

This is a bit of a flaw in the topic metadata request logic on the broker at startup. We need to fix it. Before a broker receives the first LeaderAndIsrRequest/UpdateMetadataRequest, it is technically not ready to start serving any request. But it still ends up serving TopicMetadataRequest which ca

Re: Connection reset by peer Error in Kafka server.log

Is the consumer being shutdown or interrupted? Do you see any relevant errors in your consumer's logs? Thanks, Neha On Thu, Oct 31, 2013 at 9:31 PM, Banerjee, Aparup wrote: > No I am not using a LB. Infact I keep getting the same error even if my > Kafka server and consumer are on same box. The

Re: Topic creation on restart

This is probably because some metadata requests were issued to a newly started broker before the controller has propagated the metadata. Since the broker doesn't have the metadata in the cache, it thinks the topic doesn't exist and tries to create the topic. The creation will fail though since it's

Re: Throughput Questions

Kafka has a retention policy. Messages will to automatically deleted after a certain period of time. Thanks, Jun On Thu, Oct 31, 2013 at 5:33 PM, hsy...@gmail.com wrote: > Hi guys, I have some throughput questions. > > I try to test the throughput using both the High Level Consumer and Simple

Re: Connection reset by peer Error in Kafka server.log

That was my initial hunch, but consumer logs are clean. Could it be some GC stuff on consumer? Thanks, Aparup > On Oct 31, 2013, at 9:36 PM, "Neha Narkhede" wrote: > > Is the consumer being shutdown or interrupted? Do you see any relevant > errors in your consumer's logs? > > Thanks, > Neha >

Re: Topic creation on restart

Ok, So, I can safely ignore these, it sounds like. I don't see any corresponding logging around it subsequently not succeeding to actually create the topic in zk. Would it be possible for the broker to make sure all metadata is propagated, before completing the startup sequence, and taking on me

Re: SimpleConsumer cannot read KeyedMessage.

No. The simple consumer does receive some responses and can iterate the loop: for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(m_topic, m_partition)) { //handle messages } but after that, the response still returns will byte[], I can see the content, but the iterator cannot iter

Re: SimpleConsumer cannot read KeyedMessage.

I enabled gzip compression. Each topic has 10 partitions and each partition is handled by 1 simple consumer thread. All consumers stop to iterate after iterate first several responses. The responses still return with bytes, but cannot iterate. On Thu, Oct 31, 2013 at 9:59 PM, Lu Xuechao wrote:

Re: Topic creation on restart

>> Would it be possible for the broker to make sure all metadata is propagated, before completing the startup sequence, and taking on metadata requests? Yes, that is the flaw I mentioned before. We need to fix it. On Thu, Oct 31, 2013 at 9:56 PM, Jason Rosenberg wrote: > Ok, > > So, I can safe

Re: Connection reset by peer Error in Kafka server.log

Hmm, I would suspect GC to give a Connection timeout instead of Connection reset, but I could be wrong. On Thu, Oct 31, 2013 at 9:42 PM, Banerjee, Aparup wrote: > That was my initial hunch, but consumer logs are clean. Could it be some > GC stuff on consumer? > > Thanks, > Aparup > > > On Oct 31

Re: Topic creation on restart

In this case, it appears to have gone on for 104 seconds. Should it take that long? It doesn't seem to always take this long. I guess one of the drawbacks of having a large number of topics (brings out the edge cases). Filed: https://issues.apache.org/jira/browse/KAFKA- On Fri, Nov 1, 201

Re: Purgatory

Guozhang, The documentation is not very clear. Marc's response for producer purgatory makes sense. I am not entirely clear on fetch purgatory. How does broker use purgatory? Is it a temporary holding area? What happens to the messages if purge interval is exceeded in case of either/both producer an