It looks like it is some issue with backpressure as the same behavior happens
with the client library as a custom source.
Michael
> On Aug 16, 2018, at 6:59 PM, TechnoMage wrote:
>
> I have seen this in the past and running into it again.
>
> I have a kafka consumer that is not getting all th
I have seen this in the past and running into it again.
I have a kafka consumer that is not getting all the records from the topic.
Kafka conforms there are 300k messages in each partition, and flink only sees a
total of 8000 records in the source.
Kafka is 2.0, flink is 1.4.2 connector is Fli
019/flink-kafka-consumer-groupid-not-working
>
> <https://stackoverflow.com/questions/38639019/flink-kafka-consumer-groupid-not-working>
>
> From: TechnoMage [mailto:mla...@technomage.com]
> Sent: Wednesday, April 25, 2018 8:52 AM
> To: Tzu-Li (Gordon) Tai
> Cc: user
-working
From: TechnoMage [mailto:mla...@technomage.com]
Sent: Wednesday, April 25, 2018 8:52 AM
To: Tzu-Li (Gordon) Tai
Cc: user
Subject: Re: Weird Kafka Connector issue
I have another java program reading the topic to monitor the test. It receives
60,000 records on the “travel” topic, while the
Just in case it is a metrics bug, I will add a step to do my own counting in
the Flink job.
Michael
> On Apr 25, 2018, at 9:52 AM, TechnoMage wrote:
>
> I have another java program reading the topic to monitor the test. It
> receives 60,000 records on the “travel” topic, while the kafka cons
I have another java program reading the topic to monitor the test. It receives
60,000 records on the “travel” topic, while the kafka consumer only reports
4,138. That and the incongruity of the source to the maps are what seems very
weird. I have several other topics where the source is built
I have been using the kafka connector sucessfully for a while now. But, am
getting weird results in one case.
I have a test that submits 3 streams to kafka topics, and monitors them on a
separate process. The flink job has a source for each topic, and one such is
fed to 3 separate map functio
