[
https://issues.apache.org/jira/browse/KAFKA-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep G updated KAFKA-2025:
-----------------------------
Description:
In a setup where there are two consumers C1 & C2 belonging to consumer group
CG, two partitions P1 & P2; with auto-commit disabled.
An explicit commit on ConsumerConnect commits on all the consumers i.e. a
commit called by C1 commits all messages being processed by other consumers too
here C2.
Ideally C1 should be able to commit only those messages it has consumed and not
what is being processed by C2. The effect of this behavior is that; suppose C2
crashes while processing message M after C1 commits, is that message M being
processed by C2 is not available on recovery and is lost forever; and in kafka
M is marked as consumed.
I read that this would be addressed in the rewrite -
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
Any thoughts on which release this would be addressed ?. A quick response
would be greatly appreciated.
Thanks,
Pradeep
was:
In a setup where there are two consumers C1 & C2 belonging to consumer group
CG, two partitions P1 & P2; with auto-commit disabled.
An explicit commit on ConsumerConnect commits on all the consumers i.e. a
commit called by C1 commits all messages being processed by other consumers too
here C2.
Ideally C1 should be able to commit only those messages it has consumed and not
what is being processed by C2. The effect of this behavior is that; suppose C2
crashes while processing message M after C1 commits, is that message M being
processed by C2 is not available on recovery and is lost forever; and in kafka
M is marked as consumed.
I read that this would be addressed in the rewrite -
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
Any thoughts on which release this would be addressed ?. A quick response
would be greatly appreciated.
Thanks,
Pradeep
This
> In multi-consumer setup - explicit commit, commits on all partitions
> --------------------------------------------------------------------
>
> Key: KAFKA-2025
> URL: https://issues.apache.org/jira/browse/KAFKA-2025
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.2.0
> Environment: 1. Tested in Windows
> 2. Not tested on Linux
> Reporter: Pradeep G
> Assignee: Neha Narkhede
> Priority: Blocker
>
> In a setup where there are two consumers C1 & C2 belonging to consumer group
> CG, two partitions P1 & P2; with auto-commit disabled.
> An explicit commit on ConsumerConnect commits on all the consumers i.e. a
> commit called by C1 commits all messages being processed by other consumers
> too here C2.
> Ideally C1 should be able to commit only those messages it has consumed and
> not what is being processed by C2. The effect of this behavior is that;
> suppose C2 crashes while processing message M after C1 commits, is that
> message M being processed by C2 is not available on recovery and is lost
> forever; and in kafka M is marked as consumed.
> I read that this would be addressed in the rewrite -
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
> Any thoughts on which release this would be addressed ?. A quick response
> would be greatly appreciated.
> Thanks,
> Pradeep
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
