[
https://issues.apache.org/jira/browse/NIFI-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874315#comment-15874315
]
Koji Kawamura edited comment on NIFI-3189 at 2/20/17 10:31 AM:
---------------------------------------------------------------
I've done an experiment if Kafka consumer's pause/resume API can be useful to
retain consumer connection. I've put the experimentation code in my remote
branch here.
https://github.com/ijokarumawak/nifi/commit/28ba134771ec7a7e810924f655662a29662ba9bf
It seems working as expected. After back-pressure is engaged, ConsumeKafka is
not triggered for a while, then when downstream recovers from back-pressure,
ConsumeKafka resumes consuming messages without any loss or warning messages
using the same consumer instance.
If this is a right direction, I am going to add more synchronization,
start/stop processor handling, and make it work more properly with onTrigger so
that it only try to retain when there hasn't been any polling activity for some
period of time.
Any feedback would be appreciated. Thanks!
was (Author: ijokarumawak):
I've done an experiment if Kafka consumer's pause/resume API can be useful to
retain consumer connection. I've put the experimentation code in my remote
branch here.
https://github.com/ijokarumawak/nifi/commit/28ba134771ec7a7e810924f655662a29662ba9bf
It seems working as expected. After back-pressure is engaged, ConsumeKafka is
not triggered for a while, then when downstream recovers from back-pressure,
ConsumeKafka resumes consuming messages without any loss or warning messages
using the same consumer instance.
I am going to add more synchronization, start/stop processor handling, and make
it work more properly with onTrigger so that it only try to retain when there
hasn't been any polling activity for some period of time.
Any feedback would be appreciated. Thanks!
> ConsumeKafka 0.9 and 0.10 can cause consumer rebalance when backpressure is
> engaged
> -----------------------------------------------------------------------------------
>
> Key: NIFI-3189
> URL: https://issues.apache.org/jira/browse/NIFI-3189
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Joseph Witt
>
> ConsumeKafka processors can alert to rebalance issues when backpressure is
> engaged on the output connection and is then freed up. This is because we're
> not doing anything with those consumers for a period of time and the kafka
> client detects this and initiates a rebalance. We should ensure that even
> when we cannot send more data due to back pressure that we at least have some
> sort of keep alive behavior with the kafka client. Or, if that isn't an
> option we should at least document the situation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
