[
https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083551#comment-14083551
]
nicu marasoiu commented on KAFKA-1510:
--------------------------------------
Hi,
I have given it more consideration, and indeed to "||force" on 318 it is a
different concern, which can be taken to another task.
The risk which it would solve, is that when kafka is out for the limited retry
count during shutdown, at least zookeeper would get the offsets, and the
consumer will not rewind. However it is low probability that both systems are
down, so zookeeper would likely be up to date when kafka is down, for instance.
The probability that zookeeper will get flooded with all offsets multiple times
kafka is retried is comparable to that low probability.
So, for this task, I take out that line 318 part of the patch, test went fine.
I will create another task for isAutoCommit issue and analyze if the meaning is
truly reversed, cause I feel it is only partially and perhaps used correctly
with the reversed name, and it is mostly diffent thing.
I will do the config changes, no prob - switch the default offsets.storage to
Kafka and set dual.commit to true.
> Force offset commits when migrating consumer offsets from zookeeper to kafka
> ----------------------------------------------------------------------------
>
> Key: KAFKA-1510
> URL: https://issues.apache.org/jira/browse/KAFKA-1510
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Joel Koshy
> Assignee: Joel Koshy
> Labels: newbie
> Fix For: 0.8.2
>
> Attachments: forceCommitOnShutdownWhenDualCommit.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on
> dual-commit (i.e., the consumers will commit offsets to both zookeeper and
> kafka) in addition to setting offsets.storage to kafka. However, when we
> commit offsets we only commit offsets if they have changed (since the last
> commit). For low-volume topics or for topics that receive data in bursts
> offsets may not move for a long period of time. Therefore we may want to
> force the commit (even if offsets have not changed) when migrating (i.e.,
> when dual-commit is enabled) - we can add a minimum interval threshold (say
> force commit after every 10 auto-commits) as well as on rebalance and
> shutdown.
> Also, I think it is safe to switch the default for offsets.storage from
> zookeeper to kafka and set the default to dual-commit (for people who have
> not migrated yet). We have deployed this to the largest consumers at linkedin
> and have not seen any issues so far (except for the migration caveat that
> this jira will resolve).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
