Keep on getting kafka.common.OffsetOutOfRangeException: Random times

2014-08-18 Thread pradeep.simha
Hi Team, Of late I am facing strange issue w.r.t Kafka. Random times I keep on getting these strange errors while consuming the topic: kafka.common.OffsetOutOfRangeException: Request for offset 19 but we only have log segments in the range 0 to 0. Sometimes I get like this: kafka.common.Offs

Re: consumer read from specific partition

2014-08-18 Thread Josh J
> You can see an example of using the SimpleConsumer here < https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > Any suggestions on where in the code to modify the high level producer to support reading from specific partitions ? Thanks, Josh . On Thu, Aug 14, 2014 at

Subscribe me

2014-08-18 Thread Nishit Kumar

Re: consumer read from specific partition

2014-08-18 Thread Sharninder
On Mon, Aug 18, 2014 at 7:27 PM, Josh J wrote: > > You can see an example of using the SimpleConsumer here > < > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > > > > Any suggestions on where in the code to modify the high level producer to > support reading fro

Re: consumer read from specific partition

2014-08-18 Thread Josh J
Is it possible to modify and use the high level consumer so that I can ignore processing certain partitions? On Mon, Aug 18, 2014 at 5:07 PM, Sharninder wrote: > On Mon, Aug 18, 2014 at 7:27 PM, Josh J wrote: > > > > You can see an example of using the SimpleConsumer here > > < > > > > > https

Re: offset commit api

2014-08-18 Thread Weide Zhang
Thanks Joel. Do you know if I'm using consumer group string On Mon, Aug 4, 2014 at 3:16 PM, Joel Koshy wrote: > Weide, 0.8.1.1 does not support offsets storage in Kafka. The brokers > do support offset commit requests/fetches but simply forward to > ZooKeeper - you can issue the offset commit a

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

2014-08-18 Thread Guozhang Wang
Hi Pradeep, It seems your logs gets deleted due to retention policy. Could you check the config values for log.retention.bytes and log.retention.check.interval.ms? http://kafka.apache.org/documentation.html#brokerconfigs Guozhang On Mon, Aug 18, 2014 at 5:49 AM, wrote: > Hi Team, > > Of late

Re: consumer read from specific partition

2014-08-18 Thread Jonathan Weeks
One tactic that might be worth exploring is to rely on the message key to facilitate this. It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding

Re: consumer read from specific partition

2014-08-18 Thread Josh J
> One tactic that might be worth exploring is to rely on the message key to facilitate this. > It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discardin

kafka contrib/hadoop-consumer,producer

2014-08-18 Thread Harsha
Hi, I would like to know the status regarding contrib/hadoop-consumer,hadoop-producer code. Is this considered to be deprecated or there is still interest in these projects. Is casmus the current actively developed project for writing data from kafka to hdfs. If there

Re: consumer read from specific partition

2014-08-18 Thread Jonathan Weeks
The high level API gives you access to the raw Kafka event which is MessageAndMetadata, which has two pieces: key and actual message. Best Regards, -Jonathan On Aug 18, 2014, at 9:50 AM, Josh J wrote: >> One tactic that might be worth exploring is to rely on the message key to > facilitate t

Re: offset commit api

2014-08-18 Thread Joel Koshy
If you are using the high-level consumer then yes you would be under a group. If you are using the simple consumer then no. I just saw your earlier email in this thread - if you are using the simple consumer you can still use the offset commit API and set a group manually. On Mon, Aug 18, 2014 at

Re: Subscribe me

2014-08-18 Thread Neha Narkhede
Here are instructions on how to subscribe correctly - http://kafka.apache.org/contact.html On Mon, Aug 18, 2014 at 5:41 AM, Nishit Kumar wrote: > >

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

2014-08-18 Thread Neha Narkhede
Also, what version of Kafka are you using? 0.8.1.1 is the latest most stable version. On Mon, Aug 18, 2014 at 9:36 AM, Guozhang Wang wrote: > Hi Pradeep, > > It seems your logs gets deleted due to retention policy. Could you check > the config values for log.retention.bytes and > log.retention.

How to shutdown/restart kafka cluster properly

2014-08-18 Thread Ivan Balashov
Hi, Sorry if this had been answered before. Although, I couldn't find any information besides "controlled shutdown of broker", which, I believe not fully applies here. Could anyone suggest what would be the safest strategy to shut down kafka cluster? Should brokers be brought down one-by-one or s

Re: Subscribe me

2014-08-18 Thread François Langelier
Neha, I think you just reply to the mailing, so i guess Nishit didn't received your instructions. Problem should be fixed now ;) François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communicati

Using kafka in non million users environment

2014-08-18 Thread Justin Maltat
Hello, I'm managing a study to explore possibilities for migrating a monolith architecture IT to a service oriented one. That said, the company i'm working for is not a web based company, so the number of user (i.e: the load) is not the heart of the issue, but data/services diversity is. I'm trul

Re: How to shutdown/restart kafka cluster properly

2014-08-18 Thread Joel Koshy
Please go through this section: http://kafka.apache.org/documentation.html#basic_ops_restarting On Tue, Aug 19, 2014 at 01:19:34AM +0400, Ivan Balashov wrote: > Hi, > > Sorry if this had been answered before. Although, I couldn't find any > information besides "controlled shutdown of broker", wh

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

2014-08-18 Thread pradeep.simha
Hi Neha, Yes, I am using the latest version ie (0.8.1.1). Hi Guozhang, These are the values: #log.retention.bytes= 1073741824 (Yes, this was commented by default) log.retention.check.interval.ms=6 Am I doing anything wrong here? Since I haven't touched this properties file. Thanks, Prade