[
https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115374#comment-14115374
]
nicu marasoiu commented on KAFKA-1510:
--------------------------------------
The patch makes the simplest choices:
1. unfiltered commits when storage=kafka (unfiltered to both storages if the
case).
2. unfiltered retries (even if some of the offsets have already been properly
sent in previous attempts)
A way to solve point 2 in a more general context would be to account for
freshly committed offsets, not just offsets values changes when deciding to
filter an offset or not.
> Force offset commits when migrating consumer offsets from zookeeper to kafka
> ----------------------------------------------------------------------------
>
> Key: KAFKA-1510
> URL: https://issues.apache.org/jira/browse/KAFKA-1510
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Joel Koshy
> Assignee: Joel Koshy
> Labels: newbie
> Fix For: 0.8.2
>
> Attachments:
> Patch_to_push_unfiltered_offsets_to_both_Kafka_and_potentially_Zookeeper_when_Kafka_is_con.patch,
> kafka-1510.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on
> dual-commit (i.e., the consumers will commit offsets to both zookeeper and
> kafka) in addition to setting offsets.storage to kafka. However, when we
> commit offsets we only commit offsets if they have changed (since the last
> commit). For low-volume topics or for topics that receive data in bursts
> offsets may not move for a long period of time. Therefore we may want to
> force the commit (even if offsets have not changed) when migrating (i.e.,
> when dual-commit is enabled) - we can add a minimum interval threshold (say
> force commit after every 10 auto-commits) as well as on rebalance and
> shutdown.
> Also, I think it is safe to switch the default for offsets.storage from
> zookeeper to kafka and set the default to dual-commit (for people who have
> not migrated yet). We have deployed this to the largest consumers at linkedin
> and have not seen any issues so far (except for the migration caveat that
> this jira will resolve).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
