The processing does not take much time since the application reads from the
kafka and immediately puts into a in memory queue and another thread
persist these records into DB.
On Mon, Jul 9, 2018 at 12:44 PM, dev loper wrote:
> Hi Steve,
>
> When the issue occurs I could see the lag shown for t
Hi Steve,
When the issue occurs I could see the lag shown for the corresponding
partitions in the output of kafka-consumer-groups command . I will try
changing the log level to debug for kafka consumer and I will post if
anything seems to be outside the normal range
Thank you
Dev Loper
On Mon, J
Also, have you tried to enabled DEBUG level logging for KafkaConsumer? How
long does it take to process records in a single batch?
On Mon, Jul 9, 2018, 2:51 PM Sabarish Sasidharan
wrote:
> Can you try executing kafka-consumer-groups to see what it says? Are your
> log offsets increasing and the
Can you try executing kafka-consumer-groups to see what it says? Are your
log offsets increasing and the lag increasing when processing for some of
your partitions are stuck?
May be the problem is in the producer side.
Regards
Sab
On Mon, 9 Jul 2018, 11:43 am Steve Tian, wrote:
> Are you sure
Hi Steve,
Yes, I am sure my consumer was the only owner of the partitions. The
partitions are getting recovered at random intervals and there is only
single consumer running in the entire environment . But since the data is
time critical , its leading to data loss. I wanted to go deep and
under
Are you sure your consumer was still the owner of your partitions?
On Mon, Jul 9, 2018, 12:54 PM dev loper wrote:
> Hi Kafka Users,
>
> I am desperate to find an answer to this issue. I would like to know
> whether my issue is due to a Single Kafka Consumer ? Where should I look
> for answers f
Hi Kafka Users,
I am desperate to find an answer to this issue. I would like to know
whether my issue is due to a Single Kafka Consumer ? Where should I look
for answers for this issue? Is it something do with the Kafka Broker ? I
am using Kafka version 0.11.01 version for both Kafka broker an c
Hi Kafka Streams Users,
I have posted the same question on stackoverflow and if anybody could point
some directions it would be of great help.
https://stackoverflow.com/questions/51214506/kafka-consumer-hung-on-certain-partitions-single-kafka-consumer
On Fri, Jul 6, 2018 at 10:25 PM, dev loper
