No I am using 0.8.0.2 kafka. I did some experiments with changing the
parallelism from 4 to 16 now the lag has reduced to 20 min from 2 hours,
the cpu utilization (load avg) has gone up from 20-30 % to 50-60 % , so
parallelism does seem to play a role in reducing the processing lag in
flink as I e
Hi,
are you per change using Kafka 0.9?
Cheers,
Aljoscha
On Tue, 10 May 2016 at 08:37 Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> Robert,
> Regarding the event qps 4500 events/sec may not be large no, but I am
> seeing some issue in processing the events due to processing pow
Robert,
Regarding the event qps 4500 events/sec may not be large no, but I am
seeing some issue in processing the events due to processing power that I
am using, I have deployed flink app on 3 node yarn cluster one node is a
master, 2 slave nodes which has the taskmanager running. Each machine is
Robert, what do you think about adding a note about this to the Kafka
consumer docs? This has come up a couple of times on the mailing list
already.
– Ufuk
On Fri, May 6, 2016 at 12:07 PM, Balaji Rajagopalan
wrote:
> Thanks Robert appreciate your help.
>
> On Fri, May 6, 2016 at 3:07 PM, Robert
Thanks Robert appreciate your help.
On Fri, May 6, 2016 at 3:07 PM, Robert Metzger wrote:
> Hi,
>
> yes, you can use Kafka's configuration setting for that. Its called
> "auto.offset.reset". Setting it to "latest" will change the restart
> behavior to the current offset ("earliest" is the opposi
Hi,
yes, you can use Kafka's configuration setting for that. Its called
"auto.offset.reset". Setting it to "latest" will change the restart
behavior to the current offset ("earliest" is the opposite).
How heavy is the processing you are doing? 4500 events/second sounds not
like a lot of throughpu
