GitHub user guozhangwang opened a pull request:
https://github.com/apache/kafka/pull/3806
KAFKA-5797: Handle metadata not available in store registration
This is the backport of #3748 for trunk
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guozhangwang/kafka
K5797-handle-metadata-available-0110
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3806.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3806
----
commit d7454306533c3be5edd52415c17f4f088cff6741
Author: Guozhang Wang <[email protected]>
Date: 2017-08-30T15:43:22Z
KAFKA-5797: Delay checking of partition existence in StoreChangelogReader
1. Remove timeout-based validatePartitionExists from StoreChangelogReader;
instead only try to refresh metadata once after all tasks have been created and
their topology initialized (hence all stores have been registered).
2. Add the logic to refresh partition metadata at the end of initialization
if some restorers needing initialization cannot find their changelogs, hoping
that in the next run loop these stores can find their changelogs.
As a result, after `initialize` is called we may not be able to start
initializing all the `needsInitializing` ones.
As an optimization, we would not call `consumer#partitionsFor` any more,
but only `consumer#listTopics` fetching all the topic metadata; so the only
blocking calls left are `listTopics` and `endOffsets`, and we always capture
timeout exceptions around these two calls, and delay to retry in the next run
loop after refreshing the metadata. By doing this we can also reduce the number
of request round trips between consumer and brokers.
Author: Guozhang Wang <[email protected]>
Reviewers: Damian Guy <[email protected]>, Matthias J. Sax
<[email protected]>
Closes #3748 from guozhangwang/K5797-handle-metadata-available
commit 871d3e233f97a0bf6f9e3ae487f00a128276fe42
Author: Guozhang Wang <[email protected]>
Date: 2017-09-06T07:03:16Z
minor fixes
commit 914880c55452d41161de8388553cc69ed9b6c5e0
Author: Guozhang Wang <[email protected]>
Date: 2017-09-07T05:34:10Z
Merge branch '0.11.0' of https://git-wip-us.apache.org/repos/asf/kafka into
K5797-handle-metadata-available-0110
commit 25541c4c8e629401caea23749c2745937cf5a439
Author: Guozhang Wang <[email protected]>
Date: 2017-09-07T05:50:45Z
fix some unit tests
commit 360ba8cb30ea62134c0d5301c1f39ea494a51bcd
Author: Guozhang Wang <[email protected]>
Date: 2017-09-07T06:33:54Z
fix unit tests
----
---
