That seems likely. I'd try either catching the exception and resetting the
offset, or upping log.retention.hours. I'd be interested in hearing if that
fixes the problem.
-Steve
> On Aug 19, 2014, at 11:54 PM, <pradeep.si...@wipro.com> wrote:
>
> Thank you for your reply. Oh is retention hours have affect on this? I didn't
> knew this. I have log.retention.hours set to 1, and during development we
> test this once a 15 mins or hour or 2. So do you think this is causing the
> issue?
>
> Thanks,
> Pradeep Simha
> Technical Lead
>
> -----Original Message-----
> From: Steve Miller [mailto:st...@idrathernotsay.com]
> Sent: Tuesday, August 19, 2014 6:13 PM
> To: users@kafka.apache.org
> Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random
> times
>
> Also, what do you have log.retention.hours set to? How often do you
> publish messages?
>
> I can envision a scenario in which you don't publish to a topic often, and
> in fact publish so infrequently that everything in the topic ages out from
> log.retention.hours first.
>
> I don't know exactly what happens should that occur, but I've seen some
> stuff that makes me think that the offsets might go back to zero -- or maybe
> they do if the broker restarts, so you might check to be sure that's not
> happening.
>
> From what I've seen in that regard, I've been wondering if part of the way
> most long-running Kafka consumers shouldn't be designed is to catch that
> exception and either set their offset to the first available message or the
> last available message, depending on whether their priority is to get every
> message or if it's to get the most recent messages. Though in that scenario
> maybe it's that the first and last messages are the same by definition since
> there aren't any messages left in the topic. (-:
>
> It's also possible that the specific topic weirdness that my specific
> installation has been running into is causing that and it only happens for
> the stuff I work on, so definitely take this with a grain of salt, I'm no
> expert, just relating the local folklore.
>
> -Steve
>
>> On Tue, Aug 19, 2014 at 09:12:30AM +0000, pradeep.si...@wipro.com wrote:
>> Hi Team,
>>
>> Can someone please help me in this? This is really becoming road block to
>> our project we should decide whether to continue to use Kafka or some other
>> project as it is becoming too much of unstable.
>>
>> Thanks,
>> Pradeep Simha
>> Technical Lead
>>
>> -----Original Message-----
>> From: pradeep.si...@wipro.com [mailto:pradeep.si...@wipro.com]
>> Sent: Tuesday, August 19, 2014 9:30 AM
>> To: users@kafka.apache.org
>> Subject: RE: Keep on getting kafka.common.OffsetOutOfRangeException:
>> Random times
>>
>> Hi Neha,
>>
>> Yes, I am using the latest version ie (0.8.1.1).
>>
>> Hi Guozhang,
>>
>> These are the values:
>>
>> #log.retention.bytes= 1073741824 (Yes, this was commented by default)
>>
>> log.retention.check.interval.ms=60000
>>
>> Am I doing anything wrong here? Since I haven't touched this properties file.
>>
>> Thanks,
>> Pradeep Simha
>> Technical Lead
>>
>> -----Original Message-----
>> From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
>> Sent: Tuesday, August 19, 2014 2:27 AM
>> To: users@kafka.apache.org
>> Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException:
>> Random times
>>
>> Also, what version of Kafka are you using? 0.8.1.1 is the latest most stable
>> version.
>>
>>
>>> On Mon, Aug 18, 2014 at 9:36 AM, Guozhang Wang <wangg...@gmail.com> wrote:
>>>
>>> Hi Pradeep,
>>>
>>> It seems your logs gets deleted due to retention policy. Could you
>>> check the config values for log.retention.bytes and
>>> log.retention.check.interval.ms?
>>>
>>> http://kafka.apache.org/documentation.html#brokerconfigs
>>>
>>> Guozhang
>>>
>>>
>>>> On Mon, Aug 18, 2014 at 5:49 AM, <pradeep.si...@wipro.com> wrote:
>>>>
>>>> Hi Team,
>>>>
>>>> Of late I am facing strange issue w.r.t Kafka. Random times I keep
>>>> on getting these strange errors while consuming the topic:
>>>>
>>>>
>>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but
>>>> we only have log segments in the range 0 to 0.
>>>> Sometimes I get like this:
>>>>
>>>>
>>>> kafka.common.OffsetOutOfRangeException: Request for offset 19 but
>>>> we only have log segments in the range 19 to 22.
>>>>
>>>> That number keeps on changing (with random ranges). I don't know
>>>> what is the problem here. Both producer and consumer will work
>>>> perfectly, but I keep on getting these errors randomly. In that
>>>> situation if I clear the logs, remove the broker again it starts working
>>>> fine again.
>>>>
>>>> Can anyone please help me in this regard? This is affecting our
>>>> application stability, if any more information required I can
>>>> provide,
>>> also
>>>> we are using only the defaults provided by the kafka we didn't
>>>> changed
>>> any
>>>> settings.
>>>>
>>>> Thanks,
>>>> Pradeep Simha
>>>> Technical Lead
>>>>
>>>> The information contained in this electronic message and any
>>>> attachments to this message are intended for the exclusive use of
>>>> the addressee(s)
>>> and
>>>> may contain proprietary, confidential or privileged information.
>>>> If you
>>> are
>>>> not the intended recipient, you should not disseminate, distribute
>>>> or
>>> copy
>>>> this e-mail. Please notify the sender immediately and destroy all
>>>> copies
>>> of
>>>> this message and any attachments.
>>>>
>>>> WARNING: Computer viruses can be transmitted via email. The
>>>> recipient should check this email and any attachments for the presence of
>>>> viruses.
>>>> The company accepts no liability for any damage caused by any
>>>> virus transmitted by this email.
>>>>
>>>> www.wipro.com
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>
>> The information contained in this electronic message and any attachments to
>> this message are intended for the exclusive use of the addressee(s) and may
>> contain proprietary, confidential or privileged information. If you are not
>> the intended recipient, you should not disseminate, distribute or copy this
>> e-mail. Please notify the sender immediately and destroy all copies of this
>> message and any attachments.
>>
>> WARNING: Computer viruses can be transmitted via email. The recipient should
>> check this email and any attachments for the presence of viruses. The
>> company accepts no liability for any damage caused by any virus transmitted
>> by this email.
>>
>> www.wipro.com
>>
>> The information contained in this electronic message and any attachments to
>> this message are intended for the exclusive use of the addressee(s) and may
>> contain proprietary, confidential or privileged information. If you are not
>> the intended recipient, you should not disseminate, distribute or copy this
>> e-mail. Please notify the sender immediately and destroy all copies of this
>> message and any attachments.
>>
>> WARNING: Computer viruses can be transmitted via email. The recipient should
>> check this email and any attachments for the presence of viruses. The
>> company accepts no liability for any damage caused by any virus transmitted
>> by this email.
>>
>> www.wipro.com
>
> The information contained in this electronic message and any attachments to
> this message are intended for the exclusive use of the addressee(s) and may
> contain proprietary, confidential or privileged information. If you are not
> the intended recipient, you should not disseminate, distribute or copy this
> e-mail. Please notify the sender immediately and destroy all copies of this
> message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient should
> check this email and any attachments for the presence of viruses. The company
> accepts no liability for any damage caused by any virus transmitted by this
> email.
>
> www.wipro.com
