Pradeep, The simple consumer will throw exception upon getting this response by design, for the high-level consumer it will automatically re-set its offset according to the reset policy (either latest or earliest).
Guozhang On Wed, Aug 20, 2014 at 1:40 AM, <pradeep.si...@wipro.com> wrote: > Sure, I would try with setting longer retention hours. But I feel this > would not be good approach? Should we raise it as a bug? > > Thanks, > Pradeep Simha > Technical Lead > > -----Original Message----- > From: Manjunath Shivakumar [mailto:manjunath.shivaku...@betfair.com] > Sent: Wednesday, August 20, 2014 1:31 PM > To: users@kafka.apache.org > Subject: RE: Keep on getting kafka.common.OffsetOutOfRangeException: > Random times > > We had a similar issue in our dev environments, where we had to configure > aggressive log retention to save space. > And the clients kept failing with this error, on Mondays, because the > message from friday had got deleted. > Perhaps compaction would help in this scenario too? > https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction > > ________________________________________ > From: Steve Miller [st...@idrathernotsay.com] > Sent: 20 August 2014 08:47 > To: users@kafka.apache.org > Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: > Random times > > That seems likely. I'd try either catching the exception and resetting > the offset, or upping log.retention.hours. I'd be interested in hearing if > that fixes the problem. > > -Steve > > > On Aug 19, 2014, at 11:54 PM, <pradeep.si...@wipro.com> wrote: > > > > Thank you for your reply. Oh is retention hours have affect on this? I > didn't knew this. I have log.retention.hours set to 1, and during > development we test this once a 15 mins or hour or 2. So do you think this > is causing the issue? > > > > Thanks, > > Pradeep Simha > > Technical Lead > > > > -----Original Message----- > > From: Steve Miller [mailto:st...@idrathernotsay.com] > > Sent: Tuesday, August 19, 2014 6:13 PM > > To: users@kafka.apache.org > > Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: > > Random times > > > > Also, what do you have log.retention.hours set to? How often do you > publish messages? > > > > I can envision a scenario in which you don't publish to a topic often, > and in fact publish so infrequently that everything in the topic ages out > from log.retention.hours first. > > > > I don't know exactly what happens should that occur, but I've seen > some stuff that makes me think that the offsets might go back to zero -- or > maybe they do if the broker restarts, so you might check to be sure that's > not happening. > > > > From what I've seen in that regard, I've been wondering if part of the > way most long-running Kafka consumers shouldn't be designed is to catch > that exception and either set their offset to the first available message > or the last available message, depending on whether their priority is to > get every message or if it's to get the most recent messages. Though in > that scenario maybe it's that the first and last messages are the same by > definition since there aren't any messages left in the topic. (-: > > > > It's also possible that the specific topic weirdness that my specific > installation has been running into is causing that and it only happens for > the stuff I work on, so definitely take this with a grain of salt, I'm no > expert, just relating the local folklore. > > > > -Steve > > > >> On Tue, Aug 19, 2014 at 09:12:30AM +0000, pradeep.si...@wipro.com > wrote: > >> Hi Team, > >> > >> Can someone please help me in this? This is really becoming road block > to our project we should decide whether to continue to use Kafka or some > other project as it is becoming too much of unstable. > >> > >> Thanks, > >> Pradeep Simha > >> Technical Lead > >> > >> -----Original Message----- > >> From: pradeep.si...@wipro.com [mailto:pradeep.si...@wipro.com] > >> Sent: Tuesday, August 19, 2014 9:30 AM > >> To: users@kafka.apache.org > >> Subject: RE: Keep on getting kafka.common.OffsetOutOfRangeException: > >> Random times > >> > >> Hi Neha, > >> > >> Yes, I am using the latest version ie (0.8.1.1). > >> > >> Hi Guozhang, > >> > >> These are the values: > >> > >> #log.retention.bytes= 1073741824 (Yes, this was commented by default) > >> > >> log.retention.check.interval.ms=60000 > >> > >> Am I doing anything wrong here? Since I haven't touched this properties > file. > >> > >> Thanks, > >> Pradeep Simha > >> Technical Lead > >> > >> -----Original Message----- > >> From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > >> Sent: Tuesday, August 19, 2014 2:27 AM > >> To: users@kafka.apache.org > >> Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: > >> Random times > >> > >> Also, what version of Kafka are you using? 0.8.1.1 is the latest most > stable version. > >> > >> > >>> On Mon, Aug 18, 2014 at 9:36 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > >>> > >>> Hi Pradeep, > >>> > >>> It seems your logs gets deleted due to retention policy. Could you > >>> check the config values for log.retention.bytes and > >>> log.retention.check.interval.ms? > >>> > >>> http://kafka.apache.org/documentation.html#brokerconfigs > >>> > >>> Guozhang > >>> > >>> > >>>> On Mon, Aug 18, 2014 at 5:49 AM, <pradeep.si...@wipro.com> wrote: > >>>> > >>>> Hi Team, > >>>> > >>>> Of late I am facing strange issue w.r.t Kafka. Random times I keep > >>>> on getting these strange errors while consuming the topic: > >>>> > >>>> > >>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but > >>>> we only have log segments in the range 0 to 0. > >>>> Sometimes I get like this: > >>>> > >>>> > >>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but > >>>> we only have log segments in the range 19 to 22. > >>>> > >>>> That number keeps on changing (with random ranges). I don't know > >>>> what is the problem here. Both producer and consumer will work > >>>> perfectly, but I keep on getting these errors randomly. In that > >>>> situation if I clear the logs, remove the broker again it starts > working fine again. > >>>> > >>>> Can anyone please help me in this regard? This is affecting our > >>>> application stability, if any more information required I can > >>>> provide, > >>> also > >>>> we are using only the defaults provided by the kafka we didn't > >>>> changed > >>> any > >>>> settings. > >>>> > >>>> Thanks, > >>>> Pradeep Simha > >>>> Technical Lead > >>>> > >>>> The information contained in this electronic message and any > >>>> attachments to this message are intended for the exclusive use of > >>>> the addressee(s) > >>> and > >>>> may contain proprietary, confidential or privileged information. > >>>> If you > >>> are > >>>> not the intended recipient, you should not disseminate, distribute > >>>> or > >>> copy > >>>> this e-mail. Please notify the sender immediately and destroy all > >>>> copies > >>> of > >>>> this message and any attachments. > >>>> > >>>> WARNING: Computer viruses can be transmitted via email. The > >>>> recipient should check this email and any attachments for the > presence of viruses. > >>>> The company accepts no liability for any damage caused by any virus > >>>> transmitted by this email. > >>>> > >>>> www.wipro.com > >>> > >>> > >>> > >>> -- > >>> -- Guozhang > >> > >> The information contained in this electronic message and any > attachments to this message are intended for the exclusive use of the > addressee(s) and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately and destroy all copies of this message and any attachments. > >> > >> WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > >> > >> www.wipro.com > >> > >> The information contained in this electronic message and any > attachments to this message are intended for the exclusive use of the > addressee(s) and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately and destroy all copies of this message and any attachments. > >> > >> WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > >> > >> www.wipro.com > > > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. > > > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > > > > www.wipro.com > > ________________________________________________________________________ > In order to protect our email recipients, Betfair Group use SkyScan from > MessageLabs to scan all Incoming and Outgoing mail for viruses. > > ________________________________________________________________________ > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > > www.wipro.com > -- -- Guozhang
