Robert, what do you think about adding a note about this to the Kafka
consumer docs? This has come up a couple of times on the mailing list
already.
– Ufuk
On Fri, May 6, 2016 at 12:07 PM, Balaji Rajagopalan
<balaji.rajagopa...@olacabs.com> wrote:
> Thanks Robert appreciate your help.
>
> On Fri, May 6, 2016 at 3:07 PM, Robert Metzger <rmetz...@apache.org> wrote:
>>
>> Hi,
>>
>> yes, you can use Kafka's configuration setting for that. Its called
>> "auto.offset.reset". Setting it to "latest" will change the restart behavior
>> to the current offset ("earliest" is the opposite).
>>
>> How heavy is the processing you are doing? 4500 events/second sounds not
>> like a lot of throughput.
>>
>> On Wed, May 4, 2016 at 8:24 AM, Balaji Rajagopalan
>> <balaji.rajagopa...@olacabs.com> wrote:
>>>
>>> I am using the flink connector to read from a kafka stream, I ran into
>>> the problem where the flink job went down due to some application error, it
>>> was down for sometime, meanwhile the kafka queue was growing as expected no
>>> consumer to consume from the given group , and when I started the flink it
>>> started consuming the messages no problem so far, but consumer lag was huge
>>> since producer is a fast producer about 4500 events/sec. My question is
>>> there any flink connector configuration which can force it read from the
>>> latest offset when the flink application starts since in my application
>>> logic I do not care about older events.
>>>
>>> balaji
>>
>>
>
