It's not a bug, right? It's the way the system works (if I have been following the thread correctly) -- when the retention time passes, the message is gone. Either consume your messages sooner, or increase your retention time. Kafka is not magic, it can only do what it's told.
In practise I have found compression to be a big help -- big savings on disk space. Philip ----------------------------------------- http://www.philipotoole.com On Wednesday, August 20, 2014 1:42 AM, "pradeep.si...@wipro.com" <pradeep.si...@wipro.com> wrote: Sure, I would try with setting longer retention hours. But I feel this would not be good approach? Should we raise it as a bug? Thanks, Pradeep Simha Technical Lead -----Original Message----- From: Manjunath Shivakumar [mailto:manjunath.shivaku...@betfair.com] Sent: Wednesday, August 20, 2014 1:31 PM To: users@kafka.apache.org Subject: RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times We had a similar issue in our dev environments, where we had to configure aggressive log retention to save space. And the clients kept failing with this error, on Mondays, because the message from friday had got deleted. Perhaps compaction would help in this scenario too? https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction ________________________________________ From: Steve Miller [st...@idrathernotsay.com] Sent: 20 August 2014 08:47 To: users@kafka.apache.org Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times That seems likely. I'd try either catching the exception and resetting the offset, or upping log.retention.hours. I'd be interested in hearing if that fixes the problem. -Steve > On Aug 19, 2014, at 11:54 PM, <pradeep.si...@wipro.com> wrote: > > Thank you for your reply. Oh is retention hours have affect on this? I didn't > knew this. I have log.retention.hours set to 1, and during development we > test this once a 15 mins or hour or 2. So do you think this is causing the > issue? > > Thanks, > Pradeep Simha > Technical Lead > > -----Original Message----- > From: Steve Miller [mailto:st...@idrathernotsay.com] > Sent: Tuesday, August 19, 2014 6:13 PM > To: users@kafka.apache.org > Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: > Random times > > Also, what do you have log.retention.hours set to? How often do you >publish messages? > > I can envision a scenario in which you don't publish to a topic often, and >in fact publish so infrequently that everything in the topic ages out from >log.retention.hours first. > > I don't know exactly what happens should that occur, but I've seen some >stuff that makes me think that the offsets might go back to zero -- or maybe >they do if the broker restarts, so you might check to be sure that's not >happening. > > From what I've seen in that regard, I've been wondering if part of the way >most long-running Kafka consumers shouldn't be designed is to catch that >exception and either set their offset to the first available message or the >last available message, depending on whether their priority is to get every >message or if it's to get the most recent messages. Though in that scenario >maybe it's that the first and last messages are the same by definition since >there aren't any messages left in the topic. (-: > > It's also possible that the specific topic weirdness that my specific >installation has been running into is causing that and it only happens for the >stuff I work on, so definitely take this with a grain of salt, I'm no expert, >just relating the local folklore. > > -Steve > >> On Tue, Aug 19, 2014 at 09:12:30AM +0000, pradeep.si...@wipro.com wrote: >> Hi Team, >> >> Can someone please help me in this? This is really becoming road block to >> our project we should decide whether to continue to use Kafka or some other >> project as it is becoming too much of unstable. >> >> Thanks, >> Pradeep Simha >> Technical Lead >> >> -----Original Message----- >> From: pradeep.si...@wipro.com [mailto:pradeep.si...@wipro.com] >> Sent: Tuesday, August 19, 2014 9:30 AM >> To: users@kafka.apache.org >> Subject: RE: Keep on getting kafka.common.OffsetOutOfRangeException: >> Random times >> >> Hi Neha, >> >> Yes, I am using the latest version ie (0.8.1.1). >> >> Hi Guozhang, >> >> These are the values: >> >> #log.retention.bytes= 1073741824 (Yes, this was commented by default) >> >> log.retention.check.interval.ms=60000 >> >> Am I doing anything wrong here? Since I haven't touched this properties file. >> >> Thanks, >> Pradeep Simha >> Technical Lead >> >> -----Original Message----- >> From: Neha Narkhede [mailto:neha.narkh...@gmail.com] >> Sent: Tuesday, August 19, 2014 2:27 AM >> To: users@kafka.apache.org >> Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: >> Random times >> >> Also, what version of Kafka are you using? 0.8.1.1 is the latest most stable >> version. >> >> >>> On Mon, Aug 18, 2014 at 9:36 AM, Guozhang Wang <wangg...@gmail.com> wrote: >>> >>> Hi Pradeep, >>> >>> It seems your logs gets deleted due to retention policy. Could you >>> check the config values for log.retention.bytes and >>> log.retention.check.interval.ms? >>> >>> http://kafka.apache.org/documentation.html#brokerconfigs >>> >>> Guozhang >>> >>> >>>> On Mon, Aug 18, 2014 at 5:49 AM, <pradeep.si...@wipro.com> wrote: >>>> >>>> Hi Team, >>>> >>>> Of late I am facing strange issue w.r.t Kafka. Random times I keep >>>> on getting these strange errors while consuming the topic: >>>> >>>> >>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but >>>> we only have log segments in the range 0 to 0. >>>> Sometimes I get like this: >>>> >>>> >>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but >>>> we only have log segments in the range 19 to 22. >>>> >>>> That number keeps on changing (with random ranges). I don't know >>>> what is the problem here. Both producer and consumer will work >>>> perfectly, but I keep on getting these errors randomly. In that >>>> situation if I clear the logs, remove the broker again it starts working >>>> fine again. >>>> >>>> Can anyone please help me in this regard? This is affecting our >>>> application stability, if any more information required I can >>>> provide, >>> also >>>> we are using only the defaults provided by the kafka we didn't >>>> changed >>> any >>>> settings. >>>> >>>> Thanks, >>>> Pradeep Simha >>>> Technical Lead >>>> >>>> The information contained in this electronic message and any >>>> attachments to this message are intended for the exclusive use of >>>> the addressee(s) >>> and >>>> may contain proprietary, confidential or privileged information. >>>> If you >>> are >>>> not the intended recipient, you should not disseminate, distribute >>>> or >>> copy >>>> this e-mail. Please notify the sender immediately and destroy all >>>> copies >>> of >>>> this message and any attachments. >>>> >>>> WARNING: Computer viruses can be transmitted via email. The >>>> recipient should check this email and any attachments for the presence of >>>> viruses. >>>> The company accepts no liability for any damage caused by any virus >>>> transmitted by this email. >>>> >>>> www.wipro.com >>> >>> >>> >>> -- >>> -- Guozhang >> >> The information contained in this electronic message and any attachments to >> this message are intended for the exclusive use of the addressee(s) and may >> contain proprietary, confidential or privileged information. If you are not >> the intended recipient, you should not disseminate, distribute or copy this >> e-mail. Please notify the sender immediately and destroy all copies of this >> message and any attachments. >> >> WARNING: Computer viruses can be transmitted via email. The recipient should >> check this email and any attachments for the presence of viruses. The >> company accepts no liability for any damage caused by any virus transmitted >> by this email. >> >> www.wipro.com >> >> The information contained in this electronic message and any attachments to >> this message are intended for the exclusive use of the addressee(s) and may >> contain proprietary, confidential or privileged information. If you are not >> the intended recipient, you should not disseminate, distribute or copy this >> e-mail. Please notify the sender immediately and destroy all copies of this >> message and any attachments. >> >> WARNING: Computer viruses can be transmitted via email. The recipient should >> check this email and any attachments for the presence of viruses. The >> company accepts no liability for any damage caused by any virus transmitted >> by this email. >> >> www.wipro.com > > The information contained in this electronic message and any attachments to > this message are intended for the exclusive use of the addressee(s) and may > contain proprietary, confidential or privileged information. If you are not > the intended recipient, you should not disseminate, distribute or copy this > e-mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should > check this email and any attachments for the presence of viruses. The company > accepts no liability for any damage caused by any virus transmitted by this > email. > > www.wipro.com ________________________________________________________________________ In order to protect our email recipients, Betfair Group use SkyScan from MessageLabs to scan all Incoming and Outgoing mail for viruses. ________________________________________________________________________ The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
